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ABSTRACT 


The  purpose  of  this  thesis  is  to  design,  conduct,  and  analyze 
a  C3  conflict  simulation  that  examines  the  effect  of  information 
delay  on  mission  effectiveness.  In  the  experiment  the  outcomes  of 
student  play  and  computer  play  of  the  Tactical  Tic-Tac-Toe  (T4) 
simulation  are  analyzed  to  determine  whether  the  delay  of  certain 
types  of  information  affects  mission  performance;  and,  whether 
computer  play  results  are  similar  to  student  play  of  the  same 
scenarios.  Analysis  of  the  experimental  data  provides  strong 
evidence  that  in  both  automated  and  student  play,  delay  in 
information  about  the  direct  opponent  adversely  affects  mission 
effectiveness.  The  comparison  between  student  play  and  automated 
play  is  not  quite  as  clear.  For  the  control  teams,  which  do  not 
experience  information  delay,  automated  play  very  closely  resembles 
student  play.  But,  in  the  case  of  the  treatment  teams,  which  are 
subjected  to  information  delay,  the  evidence  is  not  strong  enough 
to  make  definitive  conclusions. 
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I.  INTRODUCTION 


A.  BACKGROUND 

On  today's  battlefield  a  military  force  must  have 
efficient  and  effective  command  and  control  to  accomplish  its 
assigned  mission.  Upon  receipt  of  the  mission,  the  commander 
utilizes  whatever  information  is  available  to  decide  how  to 
employ  the  forces.  This  course  of  action  is  transmitted  to 
subordinates  for  execution.  As  the  battle  progresses,  the 
subordinates  and  others  pass  updated  information  to  the 
commander,  who  adjusts  the  strategy  and  issues  new 
instructions  to  the  subordinates  as  appropriate.  Thus,  a 
critical  component  of  effective  command  and  control  is  the 
timely  exchange  of  accurate  and  relevant  information. 

It  is  widely  accepted  that  too  much  information,  too 
little  information,  and  delayed  information  are  all 
detrimental  to  mission  accomplishment.  But,  it  is  not  known 
with  certainty  what  constitutes  too  much  information,  too 
little  information,  or  too  long  of  a  delay  in  information 
receipt.  Nor  is  it  known  how  these  factors  vary  as  the  type 
of  mission  changes.  At  this  point  the  command,  control,  and 
communication  (C3)  functions  of  the  force  need  to  be  broken 
down  into  simpler,  more  manageable  parts  to  examine  the 
relationship  between  information  and  mission  accomplishment. 


Ideally,  this  examination  would  take  place  in  the  context  of 
actual  combat;  but  the  empirical  data  available  from  previous 
conflicts  is  typically  inadequate  to  answer  the  C3  questions 
at  hand,  and  we  cannot  impose  experiments  on  commanders  when 
new  conflicts  do  arise. 

The  alternative  is  to  study  the  issues  within  some 
abstraction  of  combat.  These  range  in  level  of  personnel 
involvement  from  large  scale  exercises  down  through  wargames 
to  computer  simulations.  Exercises  are  the  closest  to  actual 
combat,  but  experimental  conditions  are  hard  to  enforce,  and 
due  to  exercise  costs,  an  adequate  number  of  replications 
cannot  be  conducted.  Consequently,  wargames  and  computer 
simulations  are  frequently  used  to  study  C3  issues. 

War  games  and  simulations  can  be  used  effectively  for  the 
design,  training,  and  employment  of  major  C3  systems.  (Sherfey, 
1992,  p.  2)  A  war  game  is  a  representation  of  conflict  that 
uses  human  subjects  to  make  the  decisions  as  the  conflict 
progresses.  In  simulations,  decisions  are  not  made  by  human 
subjects,  they  are  made  according  to  predetermined  rules  and 
procedures;  simulations  are  often  run  on  computers .  (Sherfey, 
1992,  p.  6) 

Neither  method  is  real  conflict.  They  are  abstractions  of 
conflict  that  attempt  to  duplicate  an  actual  conflict 
situation.  However,  since  war  games  and  simulations  are  used 
in  determining  system  acquisitions  and  force  employments,  it 
is  important  that  they  imitate  actual  C3  interactions  as 
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closely  as  possible,  and  that  military  decision  makers  be  able 
to  evaluate  the  premises  of  a  simulation  when  considering  its 
implications  in  force  structuring. 

B.  PURPOSE 

The  purpose  of  this  thesis  was  to  design,  conduct,  and 
analyze  a  C3  conflict  simulation  that  examined  the  effect  of 
information  delay  on  mission  effectiveness.  In  the  experiment 
the  outcomes  of  student -play  and  machine-play  of  the  Tactical 
Tic-Tac-Toe  (T4)  simulation  were  analyzed  to  determine  whether 
the  delay  of  certain  types  of  information  affected  mission 
performance;  and,  whether  machine-play  results  were  similar  to 
those  from  student -play  of  the  same  scenarios.  The  objective 
of  the  analysis  was  to  provide  insight  into  the  types  of 
information  that  are  crucial  to  mission  accomplishment  in  a 
simple  conflict  and  to  address  the  prospect  of  computer 
emulation  of  human  decisions  in  the  C3  aspects  of  conflict. 

C.  APPROACH 

The  game  chosen  as  the  vehicle  for  the  experiment  is 
called  Tactical  Tic-Tac-Toe  (T4)  .  It  is  a  simple  war  game 
that  can  be  used  to  examine  the  effects  of  delay  of 
information  (move  knowledge) .  It  is  based  on  the  familiar 
game  of  tic-tac-toe.  It  has  many  advantages  that  make  it 
useful  for  examining  factors  that  may  affect  C3  in  conflict: 
it  is  simple  to  play  and  understand;  it  allows  the  examination 
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of  C3  concepts;  and  it  introduces  the  student  to  the  design  of 
C3  system  evaluation  experiments . 

The  experiment  was  conducted  in  two  phases.  First, 
students  played  the  semi -automated  version  of  the  game  under 
varying  combinations  of  game  parameters.  Their  mission 
effectiveness  was  determined  by  their  score.  The  game 
parameters  were  established  to  determine  if  information  delay 
affects  mission  effectiveness.  Next,  the  same  game  scenarios 
were  played  entirely  by  a  computer.  The  computer  simulated 
the  play  of  four  individual  players  and  mission  effectiveness 
was  scored.  The  computer  play  was  compared  to  the  student 
play  to  determine  if  computer  play  satisfactorily  reproduced 
student  play. 

D.  TACTICAL  TIC-TAC-TOE  (T4) 

T4  was  developed  by  Mr.  Gary  Porter,  a  scientist  with  a 
civilian  research  corporation.  It  is  a  very  simple  and  easy 
to  learn  game  based  on  the  familiar  game  of  tic-tac-toe.  T4 
is  played  by  two  two-person  teams  on  a  game  board  that 
consists  of  two  tic-tac-toe  boards  placed  side  by  side  as 
shown  in  Figure  1. 
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Figure  1.  Tactical  Tic-Tac-Toe  Game  Board 

Teams  are  designated  X  or  0,  and  each  team  takes  one  side 
(left  or  right)  of  the  game  board.  The  game  allows  tic-tac- 
toes  to  be  made  on  each  side  of  the  game  board  and  across  the 
center  line  between  the  two  halves  of  the  game  board.  The 
game  is  structured  to  examine  the  inpact  of  various  C3  game 
parameters  on  mission  effectiveness.  Full  game  play 
instructions  are  included  in  Appendix  A. 

In  T4  the  parameters  can  be  altered  to  examine  their 
effect:  information  on  a  player's  moves  can  be  delayed  to 
other  players;  team  planning  time  can  be  provided  or  removed; 
move  conflicts  can  be  resolved  in  a  random  or  set  manner;  and, 
team  missions  can  require  independent  or  coordinated  play. 

T4  allows  the  delay  of  players'  information  about  their 
direct  enemy's  moves  (tactical),  information  about  their 
partner's  moves  (communication),  and  information  about  their 
partner's  opponent's  moves  (area) .  Information  can  be  delayed 
from  zero  to  nine  moves.  Additionally,  teams  can  be  given 
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time  to  plan  their  strategy  before  game  play,  and  missions  can 
be  selected  which  require  independent  action  or  a  coordinated 
attack  with  their  team  member.  Outcomes  can  be  examined  for 
the  effects  of  the  change  in  the  C3  game  parameters. 

The  game  was  first  played  at  the  Naval  Postgraduate  School 
in  a  manual  format.  Players  made  their  moves  on  sheets  of 
paper  and  gave  them  to  an  umpire  team  which  evaluated  all 
moves  on  a  master  game  board  and  then  scored  each  game  by 
hand.  This  was  extremely  complicated,  and  the  time  pressure 
that  resulted  when  the  umpire  team  administered  the  game  to  an 
entire  class  led  to  errors  in  scoring.  To  remedy  the  problem 
a  semi -automated  form  of  the  game  was  developed  which 
automatically  generated  individual  player  board  views  after 
each  move  and  scored  the  game. 

The  semi -automated  form  of  the  game  was  administered  to 
student  subjects  and  is  the  basis  of  this  thesis.  The  program 
allows  all  elements  of  mission,  delay,  and  conflict  resolution 
to  be  entered  into  a  computer-generated  master  game  board. 
The  master  game  board  shows  the  current  status  of  all  moves, 
without  the  added  delays,  to  the  umpire  team. 

In  the  play  of  the  game,  each  individual  player  receives 
a  game  board  on  which  they  make  their  initial  move. 
Individual  player  team  members  turn  in  their  hand-marked  moves 
on  their  game  board  sheet  to  the  umpire  team.  The  umpire  team 
then  enters  the  moves  onto  the  master  game  board.  The 
computer  automatically  generates  new  game  views  for  each 
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player  based  on  their  delays  and  maintains  a  "true"  master 
version  of  the  game  for  the  umpire  team.  The  updated  move 
sheet  is  passed  out  to  the  team  members  and  play  repeats  until 
the  game  is  completed.  At  the  end  of  play,  the  computer 
automatically  scores  the  game.  The  semi -automated  T4  game 
experiment  is  covered  in  detail  in  Chapter  II. 

Along  with  the  semi -automated  form  of  the  game,  a  fully 
automated  version  was  developed  in  which  all  four  team 
positions  are  played  by  a  computer.  The  computer  is 
programmed  to  play  individual  positions  according  to  different 
strategies  which  are  entered  by  the  umpire.  In  this  way  a 
player  position  can  be  given  a  predisposition  to  play  certain 
games  with  particular  strategies:  defensively;  offensively; 
randomly;  or  at  some  point  in-between. 

The  type  of  strategy  is  varied  by  the  weighting  the  umpire 
gives  certain  board  patterns  or  block  positions.  Board 
patterns  are  the  different  combinations  of  horizonal, 
vertical,  and  diagonal  tic-tac-toes  that  can  be  made  on  the 
left  or  right  boards  and  in  the  center  section  across  both 
boards.  Block  positions  are  the  nine  individual  move 
positions  on  each  board.  The  same  game  scenarios  were  used 
for  semi -automated  and  fully  automated  game  play.  The  fully 
automated  T4  game  experiment  is  covered  in  detail  in 
Chapter  III . 

In  examining  the  results  from  both  types  of  games,  this 
thesis  evaluates  two  areas  of  C3  simulation  using  T4: 
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•  Did  the  delay  of  certain  types  of  information  affect  a 
team's  mission  effectiveness? 

•  Did  fully  automated  game  play  of  the  same  scenarios 
produce  the  same  results  as  student  game  play? 


The  conclusions  of  the  analysis  are  presented  in  Chapter  IV. 
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IX.  T4  GAME  PLAY  BY  STUDENTS 


A.  INTRODUCTION 

Tactical  Tic-Tac-Toe  (T4)  is  a  double  game  board  version 
of  basic  tic-tac-toe.  It  is  a  simple  game  of  conflict.  The 
automated  features  of  the  game  allow  individual  player  views 
of  the  game  to  be  altered  by  delaying  the  display  of 
particular  players'  moves  on  the  game  board.  This  delay  of 
the  move  knowledge  simulates  information  delay.  Additionally, 
the  set  up  of  the  game  parameters  introduces  the  design  of  a 
C3  system  evaluation  experiment. 

The  semi -automated  form  of  T4  was  played  during  eight  2- 
hour  lab  sessions  of  CC-4003  to  provide  students  instruction 
on  the  effects  of  intelligence  on  game  results  and  to  further 
refine  the  game.  The  procedures  for  the  playing  the  game  are 
provided  in  Appendix  A. 

B.  EXPERIMENTAL  DBSXGN 

1.  Subjects 

Test  subjects  consisted  of  22  CC-4003  class  members. 
Each  team  was  composed  of  two  members .  Teams  were  scheduled 
to  randomly  play  different  scenarios  and  other  teams.  Teams 
never  played  more  than  two  games  in  succession  with  any  other 
team.  Team  members  could  generally  plan  their  strategy  prior 
to  game  start,  but  were  not  allowed  to  jointly  plan  after 


their  game  missions  were  known.  Teams  were  given  introductory 
game  plays  to  familiarize  them  with  the  game  prior  to  actual 
game  play.  A  team  was  designated  as  either  Team  X  or  Team  0. 
One  team  did  not  receive  information  delays  (delay  of  move 
knowledge) ,  and  was  designated  the  control  team.  The  other 
team  could  receive  different  types  of  information  delay  and 
was  designated  the  treatment  team.  Team  members  were  unaware 
of  their  designation. 

2 .  Apparatus 

The  game  requires  special  equipment  to  play.  Two 
Apple  Macintosh  computers  utilizing  system  6.0.5  and  two  laser 
printers  are  required  to  run  the  game  properly.  A  paper 
cutter  was  required  to  cut  the  main  game  board  into  four 
individual  user  game  pieces.  This  was  done  to  speed  the  game 
up.  HyperCard  version  2.0  is  required  along  with  the  T4  game 
software  to  run  and  score  the  game. 

The  game  was  played  in  a  room  partitioned  into  four 
bays:  three  game  bays  and  one  umpire  team  bay.  Two  teams  of 
two  players  each  were  placed  in  a  bay.  Players  were  provided 
a  scratch  sheet  with  a  copy  of  the  game  board  to  record  their 
moves  and  an  actual  game  board  piece  that  was  returned  to  the 
umpire  team  to  record  each  move  on  the  computer.  Players 
marked  their  moves  on  their  game  piece.  After  a  move  was 
marked,  the  game  pieces  were  retrieved  by  the  umpire  team  and 
returned  to  the  umpire  bay.  The  moves  were  entered  into  the 
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computer  and  a  new  game  sheet  was  printed.  Due  to  the  length 
of  time  required  to  print  four  separate  views  of  the  game,  one 
sheet  of  all  four  views  was  printed  and  cut  with  a  paper 
cutter  into  four  individual  views .  These  updated  game  pieces 
were  returned  to  the  team  for  the  next  move.  This  process 
continued  until  the  game  was  completed  and  the  score  was 
calculated  by  the  game  software. 

3 .  Procedure 

Before  the  experiment  could  be  designed,  hypotheses 
had  to  be  selected.  Many  parameters  can  be  tested  in  the  T4 
game,  but  it  was  felt  that  a  more  meaningful  experiment  could 
be  designed  by  varying  only  a  few  parameters .  The  initial 
conjecture  was  that  a  team's  performance  and  the  knowledge 
they  have  of  other  moves  being ’made  are  correlated.  The 
second  part  of  that  hypothesis  is  that  there  are  certain  types 
of  information  that  affect  a  team's  performance  more  than 
others.  Lastly,  it  was  hypothesized  that  the  correlation 
between  types  of  information  delayed  and  a  team's  mission  may 
differ;  a  team  has  that  has  a  mission  that  requires  close  team 
coordination  may  require  a  type  of  information  that  is 
different  from  that  required  by  a  team  that  has  a  mission 
requiring  more  independent  action.  With  these  assumptions  in 
mind,  the  following  experimental  hypotheses  were  developed: 

•  Teams  that  have  game  information  delayed  will  not  perform 
as  well  as  teams  with  no  information  delay. 
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•  Teams  with  coordinated  team  play  will  score  differently 
than  teams  with  independent  team  play  when  information  is 
delayed . 


To  evaluate  these  hypotheses,  the  following 
experimental  model  was  assumed: 


Yijklm  =  \L  +  TMj  +  CMj  +  TDk  +  A/C j  +  (TM*TD)  ik  +  (TM*A/C)n  + 

®ijklm 

where : 

Yijkim  =  treatment  team's  score  in  replication  m  of  setup  ijkl 
H  =  treatment  team's  mean  score  unaffected  by  parameters 
TMj  =  deviation  from  due  to  treatment  team  having 

mission  i  (i  has  two  levels:  mission  A  or  mission  B) 
CMj  =  deviation  from  ^  due  to  control  team  having  mission  j 
(j  has  two  levels:  mission  A  or  mission  B) 

TDk  =  deviation  from  ^  due  to  treatment  team  experiencing 

tactical  delay  at  level  k  (k  has  two  levels:  no  delay 
or  delay  of  1  move) 

A/C!  =  deviation  from  \l  due  to  treatment  team  experiencing 
area /communication  delay  at  level  1  (1  has  two 
levels:  no  delay  or  delay  of  1  move) 

(TM*TD)  ik  =  interaction  effect  when  treatment  team  has 
mission  i  and  tactical  delay  k 
(TM*A/C)U  =  interaction  effect  when  treatment  team  has 
mission  i  and  area/communication  delay  1 
eijkim  =  random  error  for  replicate  m  when  other  factors  are 
present  at  levels  i,  j,  k,  and  1 


To  test  the  hypothesis  using  the  model,  experimental 
design  parameters  were  established.  Teams  were  designated  as 
treatment  or  control  teams.  A  treatment  team  always  played  a 
control  team.  Control  teams  always  had  no  delay.  No 
significant  difference  between  teams  was  assumed.  The  teams 
were  all  given  introductory  games  to  familiarize  them  with 
game  play  before  the  experiment  was  started. 
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Teams  designated  as  treatment  teams  received  one  of 
two  mission  combinations:  they  could  be  assigned  a  mission  of 
victory  on  the  left  board  and  the  right  board  (victory  left/ 
right)  or  a  mission  of  victory  on  the  left  or  right  side  of 
the  board  and  victory  in  the  crossover  area  of  both  game 
boards  (victory  left/right  and  crossover)  .  Crossover  missions 
are  viewed  as  requiring  more  coordinated  teamwork  to  achieve 
the  objective.  Within  each  of  these  missions,  each  team 
member  could  have  no  delay?  tactical  delay  of  one  move;  area 
and  communication  delay  of  one  move;  or  tactical,  area,  and 
communication  delay  of  one  move. 

The  team  score  was  the  chosen  indicator  of  team 
performance.  As  discussed  above,  each  team  was  given  two 
possible  missions  and  was  scored  one  point  for  each  successful 
completion.  Ties  were  scored  as  zero.  Thus,  each  team  could 
score  either  0,  1,  or  2  points  per  game.  There  are  other 
alternative  measures  of  performance  such  as  number  of  tic-tac- 
toes  scored,  but  they  are  not  examined  in  this  thesis. 

When  two  players  chose  to  move  into  the  same 
unoccupied  game  bLock  (caused  by  intelligence  delay),  the 
conflict  was  randomly  resolved  by  the  computer.  Feedback  was 
provided  to  players  on  their  game  board  which  indicated  which 
player  received  the  block.  When  a  player  moved  into  a 
previously  occupied  block,  the  player  possessing  the  block, 
retained  it,  and  the  player  trying  to  move  there  lost  that 
move. 
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The  teams  were  allowed  to  do  general  game  planning 
before  or  after  the  game  sessions.  Once  the  game  session 
began  and  players  received  their  mission,  teams  were  not 
allowed  to  plan  specific  game  strategies  and  team  members  were 
physically  separated. 

Game  parameters  and  opposing  teams  were  randomized  to 
the  greatest  extent  possible.  This  was  done  to  avoid  teams 
becoming  to  familiar  with  a  particular  game  scenario  or 
opposing  team  and  to  control  other  factors  such  as  learning. 
Layout  patterns  for  team  randomization  are  presented  in 
Appendix  B.  The  entire  randomization  sequence  was  not 
utilized  in  the  experiment;  problems  with  game  software  and 
game  time  resulted  in  a  smaller  number  of  game  trials  than 
planned. 

Game  scenarios  were  designed  to  obtain  an  even 
distribution  of  data  points  for  each  scenario  type.  Again, 
software  problems  and  game  times  resulted  in  incomplete 
repetitions  for  different  game  scenarios.  A  listing  of  the 
game  scenarios  is  shown  in  Table  1 . 
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TABLE  1.  GAME  SCENARIOS  AND  COUNTS 


GAME 

COMBINATION 

COUNT  1 

;  A1 

Al 

2  1 

A1 

A2 

6 

Al 

A3 

6 

!  Al 

A4 

5 

|  B1 

Bl 

3 

B1 

B2 

5 

;  bi 

B3 

5 

|  Bl 

B4 

4 

1:  ai 

Bl 

6 

Al 

B2 

5 

Al 

B3 

6 

\  Al 

B4 

6 

Bl 

A2 

5 

Bl 

A3 

5 

Bl 

A4 

6 

j  TOTAL 

75 

C.  DATA 

The  T4  game  software  stores  each  round  of  game  moves  so 
that  the  entire  game  can  be  reconstructed  move  for  move.  The 
game  records  the  outcome  of  the  first  conflict  resolution  and 
scores  the  game  according  to  tic-tac-toes  and  mission 
completion.  An  example  of  raw  data  from  the  game  is  presented 
in  Appendix  C. 
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In  recording  the  games  some  data  problems  were  encountered 
which  altered  the  initial  game  design  and  later  affected  data 
analysis.  Data  problems  mostly  fell  into  two  areas:  lost 
data  due  to  lock  up  of  the  game  software  and  incomplete 
balancing  of  game  types  due  to  game  lock-up  and  unequal  game 
times.  The  differences  in  game  times  resulted  from  the 
randomization  of  the  game  play;  teams  had  to  wait  for  all 
other  teams  to  finish  game  play  to  swap  opposing  teams.  This 
increased  game  play  time  as  different  teams  and  scenarios 
required  different  amounts  of  time. 

In  recording  the  data,  the  initial  game  codes  were  altered 
to  more  easily  manipulate  the  data  during  analysis.  During 
game  design,  data  was  coded  as  follows: 

•  Mission:  A  =  victory  left  and  victory  right 

B  =  victory  left/right  and  victory  crossover 

•  Delay  type:  1  =  no  delay 

2  =  tactical  delay  only 

3  =  area/communication  delay  only 

4  =  tactical  and  area/communication  delay 

•  Mission  score:  0  =  no  mission  completion 

1  =  one  mission  completion 

2  =  both  missions  completed 

During  statistical  data  analysis,  data  was  coded  differently 
to  make  data  analysis  using  Minitab  software  easier  to 
manipulate: 

•  Mission:  1  =  victory  left  and  victory  right 

2  =  victory  left/right  and  victory  crossover 
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•  Tactical  delay:  0  =  no  delay 

1  =  delay  of  one  move 

•  Area/Comm  delay:  0  =  no  delay 

1  =  delay  of  one  move 

•  Mission  score:  Same  as  in  game  design. 

All  game  data  results  are  presented  in  Appendix  D. 

D.  RESULTS 

To  determine  the  effects  of  different  levels  of  the 
parameters  in  the  experimental  model,  data  was  analyzed  using 
the  General  Linear  Model  (GLM)  procedure  in  Minitab.  The  GLM 
is  a  regression  model  which  does  an  analysis  of  variance  using 
unbalanced  data  (different  numbers  of  readings  for  different 
cells  of  information)  .  Within  the  experimental  model,  Y  would 
be  the  game  score.  Y  is  assumed  to  be  an  additive  combination 
of  a  mean  game  score,  H,  and  the  effects  of  a  specific 
combination  of  the  parameters  on  the  score.  The  GLM  performs 
an  analysis  of  variance  to  evaluate  which,  if  any,  parameters 
cause  significant  departures  from  the  population  mean,  |l. 

Using  the  GLM  procedure,  each  parameter  was  evaluated  and 
the  resulting  p-values  were  calculated.  Parameters  with  a  p 
value  less  than  p=0.05  were  considered  significant.  The  only 
significant  parameter  using  GLM  was  the  treatment  team's 
tactical  delay  at  p=0.019.  The  entire  GLM  test  is  presented 
in  Appendix  E. 
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As  the  GLM  tests  assume  normally  distributed  data,  a  Mann- 
Whitney  test  was  performed  on  the  tactical  delay  parameter  to 
observe  if  a  difference  could  be  concluded  without  assuming  a 
normal  distribution.  The  null  hypothesis  that  the  two  medians 
were  equal  was  rejected  p=0.0451.  The  results  of  the  Mann- 
Whitney  test  are  presented  in  Appendix  E. 

An  evaluation  of  the  correlation  between  a  treatment 
team's  score  and  their  tactical  delay  was  also  performed.  A 
correlation  of  -0.262  was  calculated.  The  results  of  the 
correlation  are  presented  in  Appendix  E. 

From  the  results  of  the  data,  it  appears  that  information 
affects  mission  performance.  Team's  with  information  delay  do 
not  score  as  well  as  team's  with  no  delay.  In  particular, 
tactical  information  significantly  impacts  mission 
accomplishment;  tactical  delay  decreases  a  team's  score. 
Area/communication  information  delays  do  not  significantly 
impact  a  team's  performance.  Also,  mission  types  and 
interactions  between  parameters  do  not  significantly  affect  a 
team's  performance. 


18 


III.  T4  AUTOMATED  GAME  PLAY 


A.  INTRODUCTION 

The  second  series  of  trials  was  based  on  automated  play. 
In  automated  play,  games  moves  are  made  by  a  computer 
according  to  rules  that  are  programmed  into  the  automated  T4 
game  software.  Player  positions  can  be  programmed  to  play 
differently;  a  position  may  be  programmed  to  play  offensively, 
defensively,  or  with  some  strategy  in  between.  In  automated 
play,  the  same  scenarios  played  by  the  students  were  played  by 
the  computer.  The  object  of  the  simulation  was  to  observe  if 
the  computer  play  produced  results  similar  to  student  play. 
Game  procedures  are  presented  in  Appendix  A. 

B.  EXPERIMENTAL  DESIGN 

1.  Subjects 

The  automated  play  of  the  game  does  not  use  human 
subjects  to  play  the  game.  Instead,  a  computer  is  used  to 
simulate  the  moves  of  a  human  player.  The  computer  play  of 
the  positions  can  be  individually  varied  to  give  them  a 
predisposition  to  play  with  a  particular  strategy. 

2 .  Apparatus 

An  Apple  Macintosh  computer  was  used  to  play  the 
automated  version  of  T4.  The  software  required  is  Automated 
T4  and  HyperCard  version  2.0.  To  evaluate  the  results, 
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outputs  from  the  games  were  automatically  stored  in  Excel 
spreadsheet  software.  This  provided  easy  to  read 
documentation  of  the  game  parameters  and  results. 

3 .  Procedure 

In  designing  the  procedure  for  this  second  part  of  the 
experiment,  almost  all  of  the  design  parameters  were 
established  by  the  student  experimental  design.  The  automated 
game  design  parameters  were  expressly  kept  the  same  as  the 
student  gamed  portion  of  the  experiment. 

The  purpose  of  this  part  of  the  experiment  was  to 
evaluate  whether,  under  the  same  situations,  the  automated 
play  of  the  T4  game  by  a  computer  was  similar  to  that  of  the 
students . 

The  hypothesis  for  the  experiment  was  that  automated 
game  play  of  T4  would  provide  the  same  results  as  student  game 
play.  To  evaluate  this  hypothesis,  the  following  experimental 
model  was  assumed: 

Yijicim  =  H  +  TMj  +  CMj  +  TDk  +  A/C!  +  (TM*TD)  ik  +  (TM*A/C)  u  +  eljklm 
where : 

Yjjkim  =  treatment  team's  score  in  replication  m  of  setup  ijkl 
ll  =  treatment  team's  mean  score  unaffected  by  parameters 
TMj  =  deviation  from  ^  due  to  treatment  team  having 

mission  i  (i  has  two  levels:  mission  A  or  mission  B) 

CMj  =  deviation  from  ^  due  to  control  team  having  mission  j 
(j  has  two  levels:  mission  A  or  mission  B) 

TDk  =  deviation  from  |i  due  to  treatment  team  experiencing 

tactical  delay  at  level  k  (k  has  two  levels:  no  delay 
or  delay  of  1  move) 
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A/Cj  =  deviation  from  |i  due  to  treatment  team  experiencing 
area/communication  delay  at  level  1  (1  has  two 
levels:  no  delay  or  delay  of  1  move) 

(TM*TD)ik  =  interaction  effect  when  treatment  team  has 
mission  i  and  tactical  delay  k 
(TM*A/C)n  =  interaction  effect  when  treatment  team  has 
mission  i  and  area/communication  delay  1 
eijkim  =  random  error  for  replicate  m  when  other  factors  are 
present  at  levels  i,  j,  k,  and  1 


In  the  experimental  procedure,  as  stated  before,  the 
game  parameters  were  the  same  as  in  the  student  play  of  the 
game.  The  automated  play  of  T4,  however,  has  an  advantage 
over  the  student  play;  student  games  had  very  few  repetitions, 
but  the  automated  version  allows  the  game  to  be  played  many 
more  times.  In  the  automated  play,  each  game  scenario  was 
repeated  30  times.  Team  planning  time  and  randomization  of 
game  scenario  presentation  were  not  factors  in  the  automated 
game  play. 

Crossover  missions  were  set  up  always  victory  left 
with  crossover.  This  design  is  equivalent  to  the  student  game 
play  of  crossover  game  scenarios.  If  a  victory  left  with 
crossover  plays  a  victory  left  and  right  opponent,  the  player 
faces  an  equal  opponent  when  compared  with  a  victory  right 
with  crossover  play  against  a  victory  left  and  right  opponent. 
If  the  missions  are  victory  left  with  crossover  versus  victory- 
left  with  crossover  the  mission  is  equivalent  to  victory  right 
with  crossover  versus  victory  right  with  crossover.  No  case 
of  victory  left  with  crossover  versus  victory  right  with 
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crossover  was  played  in  the  student  play  of  the  game  and  will 
not  be  addressed  in  the  automated  play  of  the  game. 

The  team  score  was  the  chosen  indicator  of  team 
performance.  Each  automated  team  was  given  the  same  scenarios 
that  wcj  a  presented  to  the  student  teams.  Each  team  was  given 
two  possible  missions  and  one  point  was  scored  for  each 
successful  completion.  Ties  were  scored  as  zero.  Thus,  each 
team  could  score  either  0,  1,  or  2  points  per  game. 
Alternative  methods  of  scoring  were  not  used  as  it  would  be 
conducive  to  data  comparisons  with  the  student  played  game. 

There  is  one  major  difference,  however,  between  the 
automated  and  student  play  of  the  game--in  automated  play,  the 
computer  must  be  programmed  to  play  each  position  with  a 
particular  strategy  in  order  to  evaluate  its  possible  moves. 
The  software  allows  four  individual  player  positions  to  be 
programmed  with  different  game  strategies.  The  strategy  is 
introduced  by  the  preferential  weighting  of  game  patterns  and 
board  position  for  each  player.  The  weighting  system  was 
developed  by  Mr.  Porter  during  the  development  of  his 
automated  T4  game  software  and  is  covered  in  the  game 
procedures  in  Appendix  A. 

Moves  are  weighted  two  ways:  by  pattern  of  moves-- 
represented  on  the  main  programming  board;  and  by  position  of 
move--represented  on  the  small  programming  diagram  board 
beneath  the  main  board  (see  Figure  2.). 


22 


■n 

■li 

'jjjj 

mini 

H 

npi 

=.:■! 

nan 

«  ■  •  1 

m  i  ™ 

r bui • 

■■■ 

Ih  St? 

™  -  n  ifi — ■ 
*  m  tit — » 

■■■ 

■■I 

■■■ 

■■■ 

iSSE^rS 

WP* 

mu 

PH 

jin 

■1 

■1 

ill 

■■■ 

in—i™ 

: 

•  ra  mi 

■  |-J 

r:  =  ■ 

|  |  !  :l|  fU 

J=:  : 
g=:l 

m  •.  t  m — i 
- M  m  • » s  w 

■IIM. 

rmt 

iill 

■■i 

•  Ihm 

II 

H 

Figure  2.  Pattern/Position  Programming  Board 


During  the  initial  set  up  of  the  automated  game,  numbers  are 
assigned  to  tic-tac-toe  patterns  and  to  the  nine  individual 
block  positions  on  each  side  of  the  game  board.  Each  move  is 
evaluated  by  the  computer  which  adds  the  values  each  block  was 
assigned  based  on  its  location  in  a  tic-tac-toe  pattern  and 
its  location  on  the  game  board.  The  block  that  has  the 
highest  value  for  that  round  of  the  game  is  chosen  for  the 
move.  When  there  is  equal  weighting  of  a  move  based  on  the 
pattern,  position  weights  determine  the  preferential  move.  If 
there  is  equal  weighting  after  consideration  of  both  pattern 
and  position  weights,  the  computer  randomly  selects  one  of  the 
moves . 

There  is  a  wide  latitude  in  the  assignment  of 
weighting  values  for  the  individual  players  game  blocks .  The 
weighting  values  were  chosen  according  to  Mr.  Porter's 
suggested  values:  1000  represents  the  preferred  move,  and 
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500,  100,  50,  10,  and  1,  respectively,  represent  decreasing 
move  preferences. 

The  game  strategies  developed  for  the  game  were  either 
offensive  or  defensive.  In  offensive  play,  the  goal  is  to 
score  tic-tac-toes  rather  than  block  the  opponent  from  scoring 
a  tic-tac-toe.  In  defensive  play,  blocking  the  opponent  is 
preferred  to  scoring  a  tic-tac-toe.  When  a  crossover  mission 
was  selected  for  a  team,  individual  player  positions  were 
programmed  to  play  either  offensively  or  defensively  in  the 
crossover  region. 

Three  different  automated  player  strategies  were 
developed  for  the  automated  play  of  each  game  scenarios  setup: 
offensive  left/right;  offensive  left  with  crossover;  and, 
defensive  right  with  crossover.  A  description  and  weighting 
for  each  automated  player  strategy  is  shown  below.  For  ease 
of  explanation,  the  individual  player  is  assumed  to  be  an  0 
Left  player.  To  use  the  strategy  for  an  X  player,  the  X's  and 
O's  are  reversed.  To  use  the  strategy  for  a  right  player  the 
values  are  reverse  ordered  in  the  position  weighting  block.  In 
the  game  set  up,  the  control  team  which  is  the  team  that  has 
no  delay,  always  assumes  the  position  of  Team  0.  In  crossover 
missions  the  scenario  is  always  victory  left  and  crossover. 

•  Offensive  left/right  (0)  :  In  this  strategy  the  team  has 
received  a  mission  of  victory  left  and  victory  right.  The 
individual  player  is  either  the  left  player  or  the  right 
player.  The  player's  objective  is  then  to  score  as  many 
tic-tac-toes  as  possible  in  the  pattern  weighting.  The 
position  blocks  can  be  weighted  so  that  while  trying  to 
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obtain  a  tic-tac-toe  the  player  can  attempt  to  fill 
possible  crossover  blocks  in  case  the  opponent's  mission 
involves  a  crossover  mission.  As  there  is  no  crossover 
mission,  no  weighting  is  entered  for  crossover  patterns. 


Pattern  weighting: 


Position  weighting: 


Regular  Crossover 


00- 

1000 

XX- 

500 

0-- 

100 

X-- 

50 

— 

10 

ox- 

1 

Row  1:  323 
Row  2:  242 
Row  3:  323 


•  Offensive  crossover  (OC)  :  In  this  strategy  the  team  has 
received  a  mission  of  victory  left  and  victory  crossover. 
The  individual  player  is  on  the  side  chosen  for  the 
mission  with  the  crossover.  The  player  must  work  to 
achieve  tic-tac-toes  both  on  their  side  and  in  the 
crossover  area. 


Pattern  weighting: 

Position 

weighting 

Regular 

Crossover 

OO-  1000 

00-  1000 

Row  1: 

425 

XX-  500 

XX-  500 

Row  2: 

163 

0—  100 

0--  100 

Row  3 : 

425 

X—  50 

X--  50 

---  10 

—  10 

OX-  1 

OX-  1 

•  Defensive  crossover  (DC)  :  In  this  strategy  the  team  has 
received  a  mission  of  victory  left  and  crossover.  The 
player  is  on  the  right  side  and  can  contribute  to  mission 
accomplishment  through  crossover  scoring  and  by  blocking 
their  opponent's  right  side  tic-tac-toes.  Here  the  player 
must  be  the  right  side  team  player. 
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Pattern  weighting: 
Regular 


Crossover 


Position  weighting: 


00- 

500 

00-  500 

Row 

1: 

532 

XX- 

1000 

XX-  1000 

Row 

2: 

541 

0-- 

50 

0—  50 

Row 

3: 

532 

X--  100  X—  100 

—  10  —  10 

OX-  1  OX-  1 

A  defensive  left/right  player  was  not  utilized.  This 
would  describe  plays  in  which  a  player  would  not  attempt  to 
make  any  tic-tac-toes;  the  player  would  only  block  moves.  In 
a  mission  of  victory  left  and  right  this  would  be  detrimental 
to  mission  accomplishment  and  is,  therefore,  not  deemed  a 
viable  player  strategy. 

The  automated  player  strategies  used  in  each  game 
scenario  are  shown  in  Table  3 .  The  computer  pattern  and 
position  game  setups  are  presented  in  Appendix  F. 
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TABLE  3.  AUTOMATED  PLAYER  STRATEGIES 


*  GA 

COMBI] 

ME 

NATION 

PLAYERS 

OL  OR  XL  XR 

Al 

Al 

0 

0 

0 

0  II 

A1 

A2 

0 

0 

0 

o  l! 

Al 

A3 

0 

0 

0 

0  | 

Al 

A4 

0 

0 

0 

0  | 

B1 

Bl 

OC 

DC 

OC 

DC  | 

Bl 

B2 

OC 

DC 

OC 

DC 

B1 

B3 

OC 

DC 

OC 

DC 

Bl 

B4 

OC 

DC 

OC 

DC 

Al 

Bl 

0 

0 

OC 

DC 

|  Al 

B2 

0 

0 

OC 

DC 

1  A1 

B3 

0 

0 

OC 

DC 

1  A1 

B4 

0 

0 

OC 

DC 

Bl 

Al 

OC 

DC 

0 

0 

Bl 

A2 

OC 

DC 

0 

0 

Bl 

A3 

OC 

DC 

0 

0 

i: 

A4 

OC 

DC 

0 

0 

Table  coding: 

A  =  mission  of  victory  left  and  victory  right 
B  =  mission  of  victory  left/right  and  victory  crossover 

1  =  no  delays 

2  =  tactical  delay  of  one  move  (direct  opponent's  moves) 

3  =  area  and  communication  delay  of  one  move  (partner's 

moves  and  partner's  opponent's  moves) 

4  =  tactical,  area  and  communication  delay  of  one  move 
O  =  offensive  left/right  player  strategy 

OC  =  offensive  with  crossover  player  strategy 
DC  =  defensive  with  crossover  player  strategy 

C.  DATA 

There  were  no  data  problems  noted  in  the  automated  game 
play.  The  data  was  coded  like  the  student  game  scenarios. 
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During  game  play  the  computer  automatically  placed  the  game 
results  in  an  Excel  spreadsheet  file.  The  data  obtained  from 
automated  play  of  the  game  is  presented  in  Appendix  G. 

D.  RESULTS 

Like  the  student  data,  the  automated  data  was  analyzed 
using  the  General  Linear  Model (GLM)  procedure  in  Minitab. 
Each  parameter  was  evaluated  to  determine  which,  if  any, 
parameters  caused  a  significant  departure  from  the  population 
mean,  |X.  Those  parameters  with  a  p-value  of  less  than  0.05 
were  considered  significant. 

Using  the  GLM  procedure,  tactical  delay  had  a  p-value  of 
0.0  which  was  less  than  0.05.  Therefore,  tactical  delay 
exhibited  a  significant  effect  on  mission  effectiveness  in  the 
automated  game.  Additionally,  an  interaction  effect  between 
treatment  team's  mission  and  area/communication  delay  was 
significant  at  p=0.015.  The  GLM  calculations  are  presented  in 
Appendix  H . 

As  the  GLM  tests  assume  a  normal  distribution,  the  Mann- 
Whitney  test  was  performed  on  the  tactical  delay  parameter  to 
observe  if  a  difference  could  be  concluded  without  assuming  a 
normal  distribution.  The  null  hypothesis  that  the  two  medians 
were  equal  was  rejected  (p=0.00).  The  Mann-Whitney  test 
calculations  are  presented  in  Appendix  H.  The  significance  of 
the  interaction  effect  between  treatment  team  mission  and 
area/communication  delay  was  an  outcome  not  observed  in  the 
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effect  between  treatment  team  mission  and  area/communication 
delay  was  an  outcome  not  observed  in  the  student  play  of  the 
game.  To  evaluate  the  interaction,  an  interaction  plot  was 
drawn  of  the  means  of  each  parameter  combination:  mission 
type  A  with  tactical  delay;  mission  type  A  with  no  tactical 
delay;  mission  type  B  with  tactical  delay;  and  mission  type  B 
with  no  tactical  delay.  Prom  the  plot,  shown  as  Graph  1 
below,  it  can  be  observed  that  if  a  team  had  mission  A  (which 
was  victory  left  and  victory  right) ,  mission  effectiveness  was 
slightly  higher  with  tactical  delay;  if  a  team  had  mission  B 
(which  was  victory  left  and  victory  crossover)  ,  mission 
effectiveness  was  slightly  lower. 


Graph  1. 


Lastly,  the  tactical  delay  and  interaction  effects  were 
correlated  with  the  treatment  team' s  score  to  determine  if  the 
effect  was  positive  or  negative.  The  correlation  between 
treatment  team's  tactical  delay  and  treatment  team's  score  was 
-0.192.  The  correlation  between  treatment  team's  mission  and 
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which  is  due  to  the  interaction  of  the  mission  types  which 
negate  each  other's  effects. 

The  analysis  of  the  data  to  this  point  has  evaluated  what 
parameter  effects  were  observed  in  automated  play  of  the  T4 
game.  The  analysis  indicates  that  tactical  delay  decreases  a 
team's  mission  effectiveness;  and,  that  there  is  an 
interaction  effect  between  the  treatment  team's  mission  and 
their  area/communication  delay. 

Next  the  data  was  analyzed  to  determine  if  automated  T4 
game  play  emulates  student  play  of  T4.  The  data  was  first 
evaluated  at  the  broad  overview  level  by  comparing  automated 
and  student  mean  scores.  This  was  done  to  determine  whether 
the  overall  game  means  were  similar. 

Using  the  Minitab  two-sample  T-test,  the  overall  mean  of 
data  across  all  scenarios,  treatment  and  control,  in  the 
automated  game  was  found  and  compared  with  the  overall  mean  of 
data  across  all  scenarios,  treatment  and  control,  from  the 
student  played  game.  In  the  two  sample  T-test,  the  student 
game  mean  (n=150)  was  0.847  with  a  standard  deviation  of 
0.721;  the  automated  game  mean  (n=960)  was  0.972  with  a 
standard  deviation  of  0.682.  The  95  percent  confidence 
interval  for  the  automated  mean  minus  the  student  mean  was 
(0.001,  0.249)  .  The  confidence  interval  does  not  include  0.0, 
therefore,  it  can  be  rejected  that  the  two  means  are  the  same 
(p=0.048)  .  The  entire  calculation  is  presented  in  Appendix  H. 
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As  the  overall  means  were  not  similar,  the  means  of  the 
treatment  and  control  teams  were  evaluated  separately. 
Automated  and  student  means  within  each  team  type,  treatment 
and  control,  were  compared.  In  the  treatment  teams,  the 
automated  mean  was  0.812  with  a  standard  deviation  of  0.651; 
the  student  mean  was  0.667  with  a  standard  deviation  of  0.723. 
A  two  sample  T-test  was  performed,  and  the  95%  confidence 
interval  was  calculated  to  be  (-0.030,  0.322)  .  The  confidence 
interval  includes  0.0  for  the  difference  of  the  means, 
therefore  it  cannot  be  rejected  that  the  two  means  are  the 
same.  However,  the  p=0.10  is  very  close  to  the  0.05  level  and 
is  viewed  to  be  somewhat  borderline.  In  the  control  teams, 
the  automated  mean  was  1.131  with  a  standard  deviation  of 
0.676;  the  student  mean  was  1.027  with  a  standard  deviation  of 
0.677.  A  two  sample  T-test  was  performed  and  the  95% 
confidence  interval  was  calculated  to  be  (-0.062,  0.271).  The 
confidence  interval  includes  0.0  for  the  difference  of  the 
means,  therefore  it  cannot  be  rejected  that  the  two  control 
means  are  the  same  (p=0.22).  The  calculations  are  presented 
in  Appendix  H. 

To  evaluate  the  data  a  little  deeper,  a  Chi-square  test 
was  then  performed  on  the  automated  and  student  data  of  each 
team  type  (treatment  or  control)  to  evaluate  whether  the 
distributions  were  similar.  In  the  Chi-square  test  of  the 
treatment  team  scores,  the  Chi-square  value  obtained  was  8.253 
which  is  very  significant  at  p=0.0161.  The  Chi-square  test  of 


the  control  team  scores  resulted  in  a  Chi-square  value  of 
1.562  which  is  not  significant  at  p=0.4579.  The  large  Chi- 
square  value  from  the  treatment  team  indicates  that  the 
automated  and  student  distributions  are  dissimilar.  The 
control  team  distributions  have  a  very  small  Chi-square  value 
indicating  that  the  automated  and  student  distributions  are 
very  similar.  The  Chi-square  results  do  not  confirm  the 
results  of  the  T-test  on  the  treatment  team  scores,  but  the  p- 
value  of  0.10  had  forewarned  of  the  possibility  of  seeing  a 
different  result  in  other  tests.  In  the  Chi-square  test  of 
the  control  data,  the  results  did  confirm  the  T-test  outcome. 
The  Chi-square  calculations  are  presented  in  Appendix  H. 

As  there  was  some  difference  in  the  outcomes  of  the  T- 
tests  of  the  means  and  the  Chi-square  tests,  the  score 
distribution  patterns  were  graphed  to  evaluate  their 
similarities  or  differences.  Graphs  2  and  3  show  that  the 
distribution  patterns  agreed  with  the  T-test  and  Chi-square 
test — the  treatment  team  distributions  appear  dissimilar  and 
the  control  team  distributions  similar. 


Graph  2 .  Graph  3  . 
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As  both  the  means  and  score  distributions  gave  differing 
results  at  the  overview  level,  the  data  was  further  broken  out 
to  examine  the  scoring  distributions  of  automated  and  student 
games  within  individual  game  scenarios.  Using  Minitab,  the 
data  was  placed  by  scenario  in  tables  with  counts  for  each 
game  that  achieved  a  particular  mission  effectiveness  score; 
percentages  were  calculated  for  each  score  value.  The 
percentage  calculations  are  presented  in  Appendix  I. 

These  percentages  were  then  graphed  to  see  if  the 
automated  and  student  data  produced  similar  score  patterns. 
The  resulting  graphs  produced  a  mixture  of  distributions  that 
were  similar,  dissimilar,  and  ambiguous.  Graph  4  represents 
a  distribution  in  which  patterns  were  identified  as  similar. 
Graph  5  represents  a  distribution  in  which  patterns  were 
identified  as  dissimilar.  Graph  6  represents  a  distribution 
in  which  patterns  were  ambiguous .  In  reviewing  all  of  the 
scoring  patterns  for  each  individual  game  scenario,  the  graphs 
do  not  provide  conclusive  evidence  to  support  the  similarity 
or  difference  of  automated  and  student  scoring  patterns.  The 
graphs  of  all  the  distributions  are  presented  in  Appendix  J. 
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Graph  4. 


Graph  5. 


Graph  6. 


In  reviewing  the  different  analyses  of  the  data,  it 
appears  that  delay  of  tactical  information  resulted  in  a 
significant  negative  effect  on  both  student  and  automated  team 
mission  effectiveness.  Additionally,  the  automated  play 
exhibits  a  significant  interaction  effect  between  a  treatment 
team's  mission  and  their  area/communication  delay  on  mission 
effectiveness.  This  could  be  because  the  student  sample  size 
was  too  small  to  exhibit  this  effect  or  that  the  automated 
game  strategy  programming  produced  this  effect. 

When  examining  the  question  of  whether  the  automated  T4 
game  emulated  student  play,  there  is  not  conclusive  evidence 
to  support  the  hypothesis.  Again,  this  could  be  a  result  of 
the  small  student  sample  size  or  of  the  automated  game 
programming.  As  currently  programmed  for  this  experiment,  the 
automated  play  does  produce  similar  means  and  score 
distribution  patterns  in  many  scenarios,  however  the 
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differences  between  the  game  results  prevents  the  conclusion 
that  the  two  games  are  satisfactorily  similar. 
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XV.  CONCLUSIONS 


A.  SUMMARY 

The  comparison  of  student  and  automated  play  of  the  T4 
game  in  this  experiment  has  produced  the  following  results: 

•  Delay  of  tactical  information  results  in  a  significant 
negative  effect  on  student  and  automated  team  mission 
effectiveness . 

•  Automated  play  was  similar  to  student  play  when  teams  had 
no  information  delay.  When  information  delay  was  imposed 
on  a  team,  there  was  not  significant  evidence  to  conclude 
that  automated  play  did  or  did  not  satisfactorily  emulate 
student  play. 

Both  the  automated  and  student  games  illustrated  an  effect  of 
tactical  intelligence  delay  on  mission  effectiveness.  The 
differences  noted  between  student  results  and  automated 
results  was  possibly  due  to  the  small  student  sample  size 
which  did  not  provide  enough  data  to  witness  significant 
patterns  of  play  consistency,  and  by  the  automated  strategy 
programming.  The  T4  game  is  an  excellent  vehicle  for 
introducing  the  design  of  a  C3  system  evaluation.  Given  more 
consistent  student  data,  it  is  probable  that  the  automated 
strategies  can  be  programmed  to  more  closely  emulate  student 
results  and  more  conclusive  results  obtained  on  the  effects  of 
C3  parameters  on  mission  effectiveness. 
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B.  R1C0NMENDATX0NS 


In  playing  both  games,  it  became  evident  that  small 
problems  in  the  game  software  needed  to  be  fixed  to  facilitate 
game  play  and  prevent  loss  of  data.  Additionally,  as  the  game 
parameters  were  limited  to  focus  on  information  delay,  there 
remain  vast,  unexplored  combinations  of  game  parameters  for 
further  examination. 

Both  games  had  numerous  hardware  and  software  dependent 
features  about  them.  The  games  must  be  run  on  Apple  Macintosh 
computers  with  4MB  of  RAM  or  higher.  The  computers  must 
utilize  system  6. 0.5/6. 0.7.  To  run  the  T4  software,  HyperCard 
version  2.0  is  required.  These  requirements  restrict  the  use 
of  the  program.  To  obtain  wider  and  easier  use  of  T4,  the 
program  should  be  offered  in  an  IBM  compatible  format  and 
upgraded  to  utilize  the  current  Macintosh  system  7.0. 

Although  both  experiments  ran  well,  the  semi -automated 
version  occasionally  froze  up.  It  was  not  clear  why  this 
occurred  and  the  correction  of  this  problem  would  prevent  the 
loss  of  experimental  data.  There  were  no  noticeable  problems 
that  occurred  in  the  automated  play  of  the  game. 

The  semi -automated  version  of  the  game  was  a  small 
improvement  over  the  manual  version  of  the  past  in  terms  of 
number  of  runs.  Number  of  runs,  however,  does  not  address  the 
pressure  or  frustration  encountered  by  the  umpire  team  in  the 
manual  version;  here,  the  semi -automated  version  shows  a  vast 
improvement.  The  umpire  team  in  the  semi -automated  version 
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had  no  worries  about  scoring  the  game.  This  enabled  the  team 
to  concentrate  on  the  game  play  and  the  mechanics  of  moving 
all  the  teams  and  move  sheets.  The  next  step  in  the 
development  of  the  game  should  be  to  design  the  student  game 
for  play  on  a  computer  network.  This  would  allow  the  game 
board  views  to  be  quickly  transmitted  to  each  player  which 
would  speed  up  game  play  and  eliminate  the  large  amounts  of 
paper  used  playing  the  game  in  the  semi -automated  form. 

The  fully  automated  version  of  the  game  should  be  used  as 
an  introductory  experiment  for  C3  students.  The  game  allows 
students  to  independently  design  and  run  a  C3  system 
evaluation.  This  increased  involvement  provides  a  direct 
encounter  with  the  complexity  involved  in  a  seemingly  simple 
C3  system  evaluation. 

There  are  numerous  game  set  ups  that  can  be  used  in  future 
plays  of  the  game.  A  more  indicative  measure  of  performance 
could  be  the  number  of  tic-tac-toes  each  side  scores  rather 
than  just  the  mission  accomplishment  score.  Mission 
accomplishment  scores  are  very  discrete  and  do  not  show  any 
graduated  changes  in  performance  and  negates  the  effect  of 
mission  accomplishment  in  ties .  Tic-tac-toes  are  more 
numerous  and  may  show  a  more  marked  trend  when  evaluating  the 
effects  of  parameter  changes.  Another  option  is  to  develop  a 
measure  of  performance  that  is  a  weighted  combination  of  tic- 
tac-toe  and  mission  accomplishment  scores. 
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Future  games  could  increase  the  delays  to  see  if  this  more 
clearly  defines  the  effects  of  information  delays.  Planning 
strategies,  missions,  information  types,  game  sequences,  could 
all  be  examined  for  their  affects  on  a  team's  effectiveness. 

The  T4  games  provide  a  multitude  of  experimental  options 
that  should  be  examined  further  to  gain  experimental  design 
experience  in  C3  system  evaluations  and  to  further  illustrate 
the  effects  of  various  C3  parameters  on  mission  effectiveness. 


APPENDIX  A 


T4  (TACTICAL  T3) 


The  T4  game  used  in  the  OS4602  course  last  fall  was  derived  from  T3  (Tic  Tac 
Toe).  This  paper  describes  the  evolution  of  the  T4  game,  its  variants  and  its  potential 
use  as  a  simple,  but  rich  experimental  testbed  for  future  OS4602  experiments.  Also 
provided  is  a  discussion  of  a  computer  based  aid  for  use  by  the  controllers  of  the 
experiment.  Use  of  the  aid  will  result  in  more  accurate  results  and  faster  (more)  trials. 


T3  (TIC  TAC  TOE)  REVIEW 


Rules: 

•  2  Players  (X  and  0) 

•  Alternate  Turns  (X  plays  first) 

•  First  player  to  get  3  in  a  row  (TTT)  wins: 

•  Horizontal 

•  Vertical 

-  Diagonal 

•  Ties  (Cats  Games)  are  possible 

Examples: 


0 

0 

L/ 

0 

X 

X 

/ 

X 

X 

0 

z 

0 

0 

X 

Score:  X  Wins 
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Score:  Tie 


T4  BASELINE 


The  T4  baseline  game  is  like  T3  except  for  the  following: 

•  Simultaneous  moves:  Both  players  choose  their  moves  in  secret  and  then 
announce  them  simultaneously. 

•  Conflict  resolution:  The  simultaneous  move  rule  means  that  two  players  can 
move  into  the  same  cell  on  the  same  move.  A  coin  is  used  to  resolve  the  conflict.  The 
winner  of  the  coin  toss  is  awarded  the  move.  The  loser  is  not  allowed  to  make  an 
alternate  move;  thus  losing  a  turn. 

•  Scoring:  The  player  with  the  most  TTTs  wins  (not  the  first  TTT).  Therefore  the 
maximum  score  is  8  and  the  minimum  score  is  0.  Although  the  Pr(8  TTTs)  is  about  1  in 
20,000,000  (about  the  same  odds  as  winning  the  Lotto). 


Public  Board 


X  s 


Privata  Mova  Shaet 


o 


Sequence  of  Play: 

1.  Write  move  on  private  move  sheet. 

2.  Simultaneously  announce  moves. 

3.  Resolve  conflicts. 

4.  Post  on  public  board. 
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T4  Baseline  Example: 


TURN  1: 

Private  Mova  Shaat 


X 


TURN  1 A 
TURN  2l 

Prlvata  Mova  Shaat 


X 


TURN  2A 


Public  Board 
N 


S 


TURN  IB 
Public  Board 


N 


S 

TURN  2C 


TURN  n: 

Prlvata  Mova  Shaat 


X 

Laat  Turn  A 


Public  Board 


Last  Turn  B 


T4  WITH  INTELLIGENCE  DELAY 

The  intelligence  delay  game  is  like  the  T4  baseline  game  except  an  enemy 
detection  delay  factor  is  introduced.  A  player's  knowledge  of  own  moves  is  always 
real  time,  however  the  player's  knowledge  of  the  enemy's  current  move  may  be  kept 
secret  for  one  or  more  turns  depending  on  the  game  configuration  selected  before 
game  start.  For  example,  assume  player  X  is  configured  to  play  a  one  turn  intelligence 
delay  game  and  further  assume  that  two  moves  for  both  players  have  already 
occurred.  Then  Player  X  knows  of  own  moves  (X-j  and  X2)  but  is  only  provided  player 
O's  first  move  (O-j).  The  configurations  assigned  to  each  player  may  be  different.  In 
the  above  example,  player  O  might  have  been  assigned  no  intelligence  celay  and 
therefore  would  know  of  all  four  moves  (  X1(  X2  0-|  and  02) 
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Games  with  intelligence  time  delays  can  result  in  cases  where  a  rational  player 
will  move  into  a  cell  already  occupied  by  the  enemy  (but  because  of  the  delay  factor, 
the  player  is  unaware  of  the  enemy's  location).  The  conflict  is  resolved  by  awarding 
the  move  to  the  player  first  occupying  the  square  (LIFO).  Players  are  notified  of  the 
conflict  and  of  the  resolution  on  a  real  time  basis.  T4  with  intelligence  delays  requires 
maintaining  a  private  game  board  for  each  player  and  either  an  umpire  or  opponents 
who  are  good  sports. 

DOUBLE  BOARD  T4  BASELINE 

This  version  uses  two  standard  3X3  cell  T3  boards  placed  side-by-side 
designated  left  and  right  game  boards.  Thus,  the  total  game  board  is  3  X  6  cells. 
This  game  is  like  the  T4  baseline  game  except: 

•  4  Simultaneous  Moves:  Both  players  announce  their  left  and  right  moves 
simultaneously. 

•  Crossover  Scores:  A  crossover  score  is  a  TTT  that  crosses  over  the  center  line 
between  the  left  and  right  game  boards.  A  crossover  score  can  include  TTTTTT  (e.g.; 
X  wins  the  top  row  of  both  the  left  and  right  game  boards.  Diagonal  crossover  TTTs 
are  also  possible,  but  vertical  TTTs  are  not.  Multiple  TTTs  are  scored  by  counting  the 
number  of  TTTs  embedded  in  a  string  of  multiple  TTTs.  I.e; 


TTT  =  1  TTT 
TTTT  =  2  TTTs 
TTTTT  =  3  TTTs 
TTTTTT  =  4  TTTs 

The  maximum  possible  score  is  26,  minimum  score  is  0. 
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DOUBLE  BOARD  T4  WITH  INTELLIGENCE  DELAY 


This  version  combines  the  rules  of  the  single  board  T4  intelligence  delay  game 
with  the  double  board  T4  baseline  game.  Again  the  enemy  detection  delay 
configuration  can  be  set  to  real  time  (no  delay),  or  a  delay  of  one  or  more  turns.  A 
different  delay  factor  can  be  chosen  for  each  player,  for  each  side  of  the  board,  or  in 
combinations.  For  example  one  time  delay  configuration  might  allow  player  X  to 
received  real  time  intelligence  on  the  left  side  of  the  board  but  never  receive 
intelligence  on  the  right  side.  While  during  the  same  game  opponent  player  O 
receives  intelligence  delayed  by  one  turn  on  both  sides  of  the  board. 


DOUBLE  BOARD  T4  WITH  MISSIONS 

Missions  are  assigned  to  achieve  an  outcome  (victory  or  survival)  within  an 
area  of  the  game  board  (left,  right,  crossover,  or  overall).  Victory  is  achieved  by 
scoring  more  TTTs  than  the  enemy  in  the  assigned  mission  area.  Survival  is  achieved 
by  not  losing  (i.e.;  winning  or  tieing)  the  assigned  mission  area.  The  eight  mission 
assignments  are  as  follows: 

LV  s  Left  side  victory:  Score  the  most  TTTs  on  the  left  side. 

RV  s  Right  side  victory:  Score  the  most  TTTs  on  the  right  side. 

CV  =  Crossover  victory:  Score  the  most  crossover  TTTs. 

OV  =  Overall  victory:  Score  the  highest  total  TTTs  (left,  right,  and  crossovers ). 

LS  s  Left  side  survival:  don't  lose  on  the  left  side. 

RS  *  Right  side  survival:  don't  lose  on  the  right  side. 

CS  a  Crossover  survival:  don't  lose  in  the  crossover  area. 

OS  a  Overall  survival:  don't  lose  overall. 

Up  to  four  individual  missions  may  be  assigned  to  a  player's  mission  set.  Both 
players  can  be  assigned  different  mission  sets.  Scoring  is  based  on  successful 
mission  achievement  (either  by  individual  missions  or  by  mission  set).  TTTs  are  used 
to  decide  mission  outcome.  Notice  that  non-zero  sum  games  are  possible.  I.e.;  both 
sides  may  achieve  a  degree  of  success  in  some  game  configurations. 


44 


DOUBLE  BOARD  T4  WITH  MISSIONS  AND  INTELLIGENCE  DELAY 


A  combination  of  the  double  board  T4  baseline  game  with  mission  assignments 
(one  or  more  of  eight  mission  areas)  and  intelligence  delay  (real  time,  or  delays  of  one 
or  more  turns. 

TEAM  GAMES 

Team  games  consist  of  two  players  per  side:  One  player  on  a  team  is  assigned 
the  left  game  board  and  the  other  the  right.  Team  games  are  always  played  on  double 
game  boards  with  mission  assignments.  Missions  (including  mission  sets)  are 
assigned  by  team.  I.e.;  the  players  on  a  team  are  always  assigned  the  same  missions. 
However  different  mission  sets  may  be  assigned  to  the  opposing  team.  As  a  general 
rule  team  games,  especially  combinations  of  games  discussed  below,  require  an 
umpire:  Someone  who  administers  the  game  to  ensure  that  the  appropriate  amount  of 
information  is  provided  to  all  four  players.  As  discussed  below,  players  on  the  same 
team  may  be  assigned  different  delay  factors.  Thus  five  private  game  boards  are 
required:  one  for  each  player  and  the  umpire.  All  players  keep  their  game  board 
secret  from  opponents  and  their  own  partner.  The  umpire's  view  of  the  game 
represents  ground  truth  and  is  therefore  also  kept  secret.  Players  secretly  provide 
their  moves  to  the  umpire.  The  umpire  then  secretly  updates  ground  truth,  resolves 
conflicts,  and  then  secretly  returns  the  appropriate  level  of  information  to  each  player. 


TEAM  GAMES  WITH  INTELLIGENCE  DELAY 

The  intelligence  delay  factor  controls  when  players  are  provided  the  enemy's 
moves.  Intelligence  delay  factors  may  be  assigned  independently  of  player  and  game 
board  side.  Define  Tactical  intelligence  to  be  enemy  position  information  on  the 
player's  side  of  the  board,  and  Area  intelligence  to  be  enemy  position  information  on 
the  other  side  of  the  board.  Then  different  tactical  and  area  intelligence  delay  factors 
may  be  assigned  each  of  the  four  players.  E.g.;  X  left  may  be  assigned  a  tactical 
intelligence  delay  factor  of  1  turn  and  an  area  intelligence  delay  factor  of  2  turns  (X|_  = 
T-j,  A2).  The  players  partner  (XJ  and  the  enemies  (0|_,  Op)  may  each  be  assigned 
different  intelligence  delay  factors  . 
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TEAM  GAMES  WITH  COMMUNICATIONS  DELAYS 


This  version  introduces  a  communications  delay  factor  between  players  on  the 
same  team  (partners).  Communications  delay  controls  the  timeliness  of  receiving  your 
partners  move  information.  Like  intelligence  delay  factors,  communications  delays 
may  be  set  to  real  time  (no  delay)  or  a  delay  of  one  or  more  turns.  Continuing  the 
example,  if  player  X|_  is  assigned  a  communications  delay  of  one  turn  then  Xl  —  T-j  , 
A2,  C-j.  Each  of  the  four  players  may  be  assigned  different  communications  delays. 


TEAM  GAMES  WITH  FEEDBACK  DELAYS 

This  version  specifies  the  number  of  moves  each  player  must  make  before 
receiving  feedback  on  enemy  and  partner  locations.  Like  the  three  other  delay  factors 
the  feedback  delay  can  be  set  from  no  feedback  delay  to  more  than  one  turn  feedback 
delay.  In  the  no  feedback  delay  configuration,  normal  feedback  is  provided  after  each 
turn.  Where  normal  means  providing  the  level  of  information  dictated  by  the  other 
delay  factors  (tactical  intelligence,  area  intelligence,  and  communications  delays).  If 
the  delay  factor  is  greater  than  one,  then  multiple  moves  are  required  before  normal 
feedback  is  provided.  E.g.;  if  the  feedback  delay  is  set  to  2,  then  the  player  must  make 
two  moves  before  receiving  position  information  on  other  players'  moves.  In  this 
configuration,  information  when  received  is  current  to  the  second  turn  (after  adjusting 
for  intelligence  and  communications  delays).  Each  of  the  four  players  may  be 
assigned  different  feedback  delays. 


TEAM  GAMES  WITH  PLANNING 

This  version  introduces  planning  constraints  between  partners.  Planning  is  that 
part  of  the  game  that  occurs  before  the  first  move  is  executed.  There  are  three  levels 
of  planning.  Planning  levels  assigned  to  partners  must  be  the  same.  However  the 
levels  assigned  opposing  teams  may  be  different. 

•  Specific  scenario  planning  allowed.  Scenarios  are  provided  and  direct 
conversation  is  allowed  between  partners  before  game  start  in  order  to  plan  the 
specific  mission. 
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•  General  planning  allowed.  Team  members  are  assigned  and  general  planning 
sessions  are  allowed.  Planning  if  not  permitted  after  the  specific  game  scenario 
(factors)  are  assigned. 

•  Planning  not  allowed.  Players  are  assigned  to  teams  from  the  player  pool.  No 
team  conversation  is  permitted  until  the  end  of  the  game. 


EXPERIMENTAL  DESIGN  COMBINATIONS  FOR  TEAM  GAMES 

Six  factors  each  having  multiple  treatment  levels  have  been  introduced.  They  are 

•  Tactical  Intelligence  delay  (T  =  0,  1 , 2,  •  •  •  9)  turns, 

•  Area  Intelligence  delay  (A  =  0,  1 , 2,  •  •  •  9)  turns, 

•  Communications  delay  (C=  0,  1 , 2,  •  •  •  9)  turns, 

•  Feedback  delay  (F  =  0,  1 ,  2,  •  •  •  9)  turns, 

•  Planning  levels  (specific,  general,  and  none  =  Ps,  Pg,  and  Pn),  and 

•  Mission  sets  consisting  of  one  to  four  individual  missions.  E.g.;  team  X  might 
be  assigned  win  right  and  don't  lose  the  crossovers  (X  =  M|_ycs) 

The  levels  of  each  of  the  five  factors  may  be  distributed  among  players  each  game 
as  follows: 

•  Tactical  Intelligence  delay:  unrestricted  assignment  of  levels  to  players, 

•  Area  Intelligence  delay:  unrestricted  assignment  of  levels  to  players, 

•  Communications  delay:  unrestricted  assignment  of  levels  to  players, 

•  Feedback  delay:  unrestricted  assignment  of  levels  to  players, 

•  Planning  levels:  The  same  level  must  be  assigned  to  partners,  and 

•  Mission  assignments:  The  same  mission  set  must  be  assigned  to  partners. 

In  order  to  get  a  feel  for  the  magnitude  of  the  experimental  design  options  assume 
all  factors  are  limited  to  three  levels  (the  four  delay  factors  are  limited  to  real  time,  one 
turn  delay,  or  no  information  (level  9)  and  Missions  are  limited  to  one  of  left  victory, 
right  victory,  or  overall  victory  then  consider  the  following: 

•  T0.Ti,T9  with  4  players  =  24  permutations 

•  Aq,  Ai>  Ag  with  4  players  =  24  permutations 

•  Cq,  Ci,C9  with  4  players  =  24  permutations 
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•  Fq,  F-j  ,  Fg  with  4  players  =  24  permutations 

•  Ps.  Fg-  Fn  with  2  teams  =  6  permutations 

•  M  ( LV,  RV,  OV )  with  2  teams=  6  permutations 

Combining  the  above  six  factors  and  their  three  levels  by  players  yields  a  wide 
spectrum  of  experimental  design  opportunities.  Allowing  more  than  three  levels  (e.g.; 
mission  sets  from  one  to  four  missions  may  be  assigned  to  the  two  teams  where  a 
mission  set  consists  of  combinations  of  the  eight  basic  missions)  would  increase  the 
number  of  permutations  significantly. 

Experience  gained  during  the  last  T4  experiment  showed  that  while  game  play 
was  fairly  uncomplicated,  the  administration  of  the  game  by  the  controllers  was 
cumbersome,  confusing,  time  consuming,  and  error  prone.  A  computer  based  aid  for 
the  controllers  will  improve  this  facet  of  the  experiment.  Using  the  aid  will  permit 
improved  accuracy  and  allow  more  trials  during  a  given  time  period.  The  next  section 
discusses  this  computer  aid  in  depth. 
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T4  SIMULATION 


MOTIVATION 

The  initial  T4  experiment  was  conducted  as  part  of  the  OS4602 
course  in  the  fall  of  1990.  A  subset  of  the  students  were  assigned 
to  the  lead  group.  This  group  was  responsible  for  the  design, 
conduct,  analysis  and  reporting  the  results  of  the  experiment.  The 
remainder  of  the  students  served  as  subjects.  Each  trial  required 
for  subjects  assigned  to  two  teams.  Team  0  consisted  of  an  0  left 
and  right  player  while  team  X  consisted  of  a  X  left  and  X  right 
player. 

LESSONS  LEARNED 

In  retrospect  two  problem  areas  were  identified.  1.)  The  scope 
of  the  experimental  design  was  overly  ambitious  given  the  class 
time  constraints  allotted  to  the  lead  group.  The  resulting  small 
number  of  trials  pre-preempted  findings  that  were  significant.  2.) 
Data  collection  errors  were  committed  by  the  lead  group  during  the 
conduct,  scoring,  and  data  reduction  phases  of  the  experiment. 

These  errors  may  have  led  the  lead  group  to  reach  an  erroneous 
conclusion. 

AUTOMATED  T4 

A  automated  version  of  the  team  T4  game  has  been  developed  to 
assist  lead  groups  conduct  more  trials  while  hopefully  eliminating 
the  data  errors  experienced  during  the  first  T4  experiment.  The 
automated  T4  allows  the  lead  group  to  configure  game  files  based  on 
the  experimental  design  prior  to  the  conduct  of  the  trials.  During 
the  trials  the  lead  group  receives  private  moves  from  each  of  the 
four  players  and  inputs  them  into  the  automated  T4  display  by 
clicking  a  mouse  on  the  appropriate  cell  on  the  electronic  game 
board.  After  the  four  moves  are  secretly  input  by  the  lead  group,  the 
program  then  evaluates  each  players  moves  based  on  the  game 
configuration  (for  example  the  amount  and  type  of  delay)  and  then 
prints  a  new  move  sheet  for  each  player  containing  this  filtered 
information,  The  individual  move  sheets  are  returned  to  each  player 
the  next  move  cycle  begins. 

Data  for  each  game  is  automatically  collected.  The  data  includes 
the  game  configuration  data,  a  history  of  each  move  by  player,  and 
game  scoring  by  TTTs,  and  mission  areas.  The  program  also  allows 
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the  lead  group  to  write  the  results  of  the  trials  to  a  file  in  matrix 
(spreadsheet)  format  for  further  data  reduction  and  analysis. 

T4  SIMULATION 

Developing  a  T4  simulation  was  a  natural  extension  to  the 
automated  T4  game.  The  automated  T4  game  already  contained  the 
software  to  record  legal  moves,  to  collect  a  wealth  of  game  data,  to 
score  the  games,  and  to  print  the  results  to  a  data  file.  All  that  was 
required  to  complete  the  T4  simulation  was  1.)  creation  of  method 
to  generate  controlled  game  moves  based  on  predefined  user 
specified  configurations  and  2.)  to  construct  a  mechanism  to  allow 
multiple  replications  of  these  games,  once  specified  without  user 
intervention. 

Advantages  of  T4  Simulation 

The  advantages  of  using  a  T4  simulation  include  1.)  testing  the 
performance  of  the  automated  game,  2.)  analyzing  relationships 
between  T4  configuration  factors  such  as  delays,  missions 
assignments,  player  tactics,  and  initial  wins,  and  3.)  permitting 
player  versus  computer  trials  which  helps  control  the  game  play 
variables  and  increases  the  number  of  trials  (  1  human  subject  can 
play  against  3  computer  players  instead  of  three  players. 

Status  of  T4  Simulation 

The  computer  versus  computer  version  of  the  simulation  is 
complete.  Item  1.)  testing  discussed  above  has  been  completed. 

The  Item  3.)  human  versus  computer  play  version  is  not  yet 
available.  Item  2.)  analysis  of  relationships  between  game 
configurations  factors  is  the  subject  of  the  remainder  of  this  paper. 

T4  SIMULATION  OVERVIEW 

The  current  version  of  the  T4  simulation  allows  the  user  to 
configure  play  of  a  trial  including  configuration  each  of  the  four 
computer  players.  Configuration  means  specifying  such  items  as 
team  mission  assignments,  delays  associated  with  each  player,  and 
player  tactics.  The  user  also  specifies  the  number  of  replications 
desired  for  each  trial  configuration  and  which  data  file  to  save  the 
data  collected  during  the  games.  Multiple  trial  configurations  may 
be  specified  for  a  given  simulation  run.  For  example  30  replications 
of  one  game  configuration  and  30  replications  of  one  or  more 


50 


variations  to  this  configuration  may  be  specified  for  a  simulation 
run.  Once  the  user  starts  the  simulation  run  no  further  human 
intervention  is  required  until  the  replication  run  is  completed,  the 
T4  simulation  plays  the  games  according  to  the  predefined  user 
specifications  and  sends  the  results  to  user  specified  data  files. 
These  data  files  may  then  be  read  by  most  spreadsheets  or 
statistics  packages  for  further  data  reduction  under  macro  control. 
The  following  sections  discuss  the  details  of  T4  simulation  game 
configuration  including  creating  game  plans,  specification  of 
multiple  factor  levels,  description  of  the  data  that  is  automatically 
collected,  and  use  of  multiple  replications  to  simulate  T4  results. 

GAME  PLANS 

The  user  specifies  the  actions  of  the  four  computer  players  by 
creating  a  set  of  three  game  plans  for  each.  The  set  of  three  game 
plans  are  the  Regular  game  plan,  the  Crossover  game  plan,  and  the 
Cell  game  plan.  The  game  plans  enumerate  all  possible  next  plays 
for  the  player.  The  user,  by  assigning  point  values  to  these 
conditions,  controls  how  the  computer  player  plays  the  game. 

The  simulation  matches  the  actual  condition  of  the  game  board 
with  the  possible  next  moves  and  assigns  point  values  to  the  empty 
cells  (potential  next  moves).  First  the  game  board  is  matched 
against  the  regular  game  plan,  then  the  crossover  game  plan,  and 
finally  the  cell  game  plan,  the  scores  of  each  are  accumulated  in  the 
empty  cells.  The  cell  with  the  highest  point  count  is  chosen  for  the 
next  move.  Ties  are  randomly  broken.  A  more  complete  description 
of  each  of  the  three  types  of  game  plans  is  provided  below.  Also 
included  is  an  example  game  in  progress  to  illustrate  the  concepts. 

After  describing  game  plans,  the  other  user  specified  game 
configuration  are  discussed  followed  by  the  a  discussion  on  the 
automatic  data  collection  features  of  the  simulation.  We  will  then 
be  prepared  to  discuss  the  design  of  experiment  which  when 
executed  leads  to  a  better  understanding  of  the  relationships  of  the 
game  configuration  variables. 

REGULAR  GAME  PLAN 

The  regular  game  plan  regulates  the  automated  play  of  the 
computer  player  on  the  regular  (non-crossover)  portion  of  that 
player’s  side  of  the  game  board.  If  we  assume  we  are  describing 
Player  O  Left’s  game  plan  then  O  Left’s  regular  portion  of  the 
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playing  board  Is  shown  as  unshaded  in  the  figure  below. 


T4  Game  Board 


OLefi 

t  0  Right 

X  Left  X  Right 


The  simulation  first  sequentially  “looks”  at  all  eight  of  the  ways 
that  a  three  cell  regular  TTT  can  be  scored  as  shown  in  the  figure 
below. 


Eight  Regular  TTTs 


It  first  looks  at  the  3  horizontal  TTTs,  then  the  3  vertical  TTTs, 
then  the  2  diagonal  TTTs.  During  each  of  the  eight  looks  it  attempts 
to  match  the  contents  of  the  three  cells  in  the  look  with  a 
permutation  of  one  of  the  codes  in  the  regular  game  plan. 

A  regular  game  plan  is  a  table  which  enumerates  in  coded  form  all 
possible  next  moves  for  a  player  and  the  point  values  assigned  by 
the  user  to  that  move.  Player  0  Left’s  regular  game  plan  with 
sample  values  assigned  to  each  condition  is  shown  in  the  figure 
below. 


Regular  Game  Plan 
Player  O  Left 


00- 

1 .000 

XX- 

500 

0-- 

100 

X  -- 

50 

... 

1  0 

ox- 

0 

An  X  or  an  0  in  the  code  means  a  cell  has  an  X  or  an  0  in  it.  A  dash 
(-)  means  a  cell  is  blank.  As  previously  discussed,  the  program 
attempts  to  match  the  3  cells  in  the  current  look  with  permutations 
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of  the  codes  in  the  regular  game  plan.  For  example  the  simulation 
first  looks  at  the  three  cells  in  first  row  of  player  0  Left's  game 
board  and  attempts  to  match  the  contents  with  a  permutation  of  the 
first  code  “00-”  in  the  regular  game  plan.  A  match  occurs  if  the 
three  examined  cells  contain  “00-”,  “0-0",  or  “-00”.  In  other 
words,  a  match  occurs  if  the  three  cells  contain  exactly  three  X’s 
and  a  blank  in  any  order.  The  remaining  codes  are  evaluated  in  the 
same  manner  as  shown  in  the  figure  below. 

Regular  Gama  Plan  Definition 


Player  0  Left 

The  three  cells  contain 

(in  any  order)  exactly: 

OO- 

1,000 

Two  O's  and  one  empty  cell 

XX- 

500 

Two  X's  and  one  empty  cell 

0-- 

100 

One  0  and  two  empty  cells 

X  -- 

50 

One  X  and  two  empty  cells 

— 

10 

Three  empty  ceils 

ox- 

0 

One  each  0.  X.  and  empty  cell 

When  a  match  occurs  the  point  value  in  the  second  column  of  the 
game  plan  is  added  to  the  blank  cell.  Notice  that  each  code  has  at 
least  one  dash  (-)  in  it,  otherwise  a  next  move  for  that  code  would 
not  be  possible. 

The  simulation  then  steps  through  each  of  the  eight  TTT  looks, 
each  time  assigning  the  value  of  the  permuted  code  it  matches  to 
the  empty  cell.  Notice  that  if  an  empty  cell  is  in  the  center  of  the 
game  board  then  the  point  value  represents  the  sum  of  four  “looks” 
(two  diagonal,  one  horizontal,  and  one  vertical).  An  empty  corner 
cell  has  three  looks  and  an  empty  edge  cell  has  two  looks. 

Similar  procedures  are  used  to  add  the  values  in  the  crossover 
and  cell  game  plans  to  the  empty  cells.  But  first  the  following 
example  is  introduced  to  illustrated  the  procedure.  The  example 
game  is  joined  in  progress  and  the  simulation  is  evaluating  the  next 
move  for  0  left  using  the  regular  game  plan  shown  above.  Only  one 
side  of  the  game  board  is  used  to  evaluate  the  regular  game  plan:  in 
this  case  the  left  side. 
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0  Left's  Next  move 


OLef 

0  Right 

X2 

X2 

XI 

02 

01 

02 

01 

XI 

X  Left 

X  Right 

First  the  three  rows  are  evaluated.  The  top  row  (--X)  matches  a 
permutation  of  (X— )  in  the  regular  game  plan.  Therefore  50  points 
is  assigned  to  both  blank  cells  in  the  top  row.  The  middle  and 
bottom  rows  are  evaluated  in  the  same  manner  resulting  in  the 
interim  point  values  as  shown  in  the  following  figure. 


0  Left's  Next  move 


OLeft 

0  Riqh 

50 

50 

X2 

X2 

50 

XI 

50 

02 

01 

1000 

02 

01 

XI 

X  Left 

X  Right 

The  same  procedures  are  followed  in  matching  the  three  vertical 
columns  and  the  two  diagonals.  The  point  values  from  the  regular 
game  plan  is  added  to  the  values  already  in  the  empty  cell  (empty  in 
the  since  that  an  X  or  0  is  not  in  the  cell  and  thus  it  represents  a 
possible  next  move).  The  results  of  matching  and  adding  the  three 
vertical  rows  is  shown  in  the  figure  in  the  following  figure  on  the 
left.  The  point  values  found  from  matching  the  two  diagonals  is 
added  to  the  values  in  the  left  figure  and  shown  in  the  right  figure 
shown  below. 


O  Left's  Next  move 


OLeft 

0  Righ 

60 

50 

X2 

X2 

60 

XI 

50 

02 

01 

1010 

02 

01 

XI 

m 

X  Left 

X  Right 

0  Left's  Next  move 


OLeft 

0  Righ 

60 

1 

X2 

X2 

60 

XI 

50 

02 

01 

1510 

02 

01 

XI 

X  Left 

X  Right 

Based  on  the  results  of  assigning  point  values  to  empty  cells 
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based  on  the  regular  game  plan  only  the  next  move  selected  would  be 
the  lower  left  cell  (*1,510).  However  the  Crossover  and  Cell  game 
plans  also  contribute  to  the  point  values.  The  crossover  area  of  a  T4 
team  game  consists  of  all  the  cells  in  TTTs  that  cross  the 
centerline  as  shown  in  the  following  figures.  The  unshaded  portion 
of  the  figure  on  the  left  is  crossover  area.  The  twelve  crossover 
TTTs  for  player  0  Left  are  shown  on  the  right. 


Crossover  TTT  Area 


OLeft 

0  Righ 

MHS 

mm 

■hb 

X  Left 

X  Right 

Notice  that  while  crossover  TTTs  by  definition  extend  to  the  right 
side  of  the  game  board,  the  only  points  of  interest  to  0  Left  are 
those  on  the  left  side  of  the  board.  Points  on  the  right  side  of  the 
game  board  are  accumulated  using  the  game  plans  for  the  O  Right  and 
X  Right  players.  The  Crossover  game  plan  used  for  this  player  O  Left 
example  is  shown  below. 


Crossover  Game  Plan 


00- 

1 ,000 

XX- 

500 

0-- 

100 

X-- 

50 

— 

1  0 

ox- 

0 

The  procedures  for  adding  point  values  to  the  empty  (non  X  or  0) 
cells  after  accounting  for  the  topology  of  the  12  crossover  TTTs  are 
the  same  as  those  used  to  calculate  the  regular  game  plan  points. 
Each  of  the  three  cells  in  the  twelve  crossover  TTTs  are  matched 
with  the  crossover  game  plan  and  empty  cells  on  the  left  side  of  the 
board  are  assigned  point  values  as  before. 

The  below  figure  on  the  left  shows  the  point  totals  after  the 
points  associated  with  the  three  left-most  horizontal  "looks”  have 
been  matched  and  accumulated  in  the  empty  cells.  The  figure  on  the 
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right  adds  the  points  from  the  right-most  horizontal  “looks”  to  the 
accumulated  totals 


O  Left's  Next  move 


OLeft 

0  Righ 

60 

100 

X2 

X2 

60 

XI 

50 

02 

01 

1510 

02 

Ol 

XI 

X  Left 

X  Right 

O  Left's  Next  move 


OLeft 

0  Right 

60 

100 

X2 

X2 

60 

XI 

1050 

02 

01 

1510 

02 

01 

Xi 

X  Left 

X  Right 

The  point  totals  after  adding  the  remainder  of  the  points  based  on 
the  vertical  and  diagonal  looks  are  as  shown  below.  Note  that  the 
two  vertical  looks  resulted  in  no  points  added  to  cells  because  they 
were  a  permutation  of  (XO-),  And  no  points  were  added  to  empty 
cells  based  on  the  right  most  two  diagonal  looks  because  there 
weren’t  any  empty  cells. 


O  Left's  Next  move 


OLeft 

0  Riqh 

60 

100 

X? 

X2 

60 

XI 

1160 

02 

01 

1510 

02 

01 

XI 

X  Left 

X  Right 

Finally  the  points  associated  with  the  Cell  Move  Plan  are  added  in. 
The  points  in  the  example  player  O  Left  Cell  move  plan  shown  in  the 
figure  below  left  are  simply  transferred  directly  to  any  empty  cells. 
It  may  appear  that  the  points  assigned  to  the  Cell  game  plan  in  this 
example  are  too  low  to  have  an  effect  on  the  outcome.  However  they 
may  serve  as  a  tie  breaker.  The  resultant  final  score  is  shown  on 
the  right. 


Cell  Game  Plan 
Player  O  Left 


2 

3 

5 

1 

4 

2 

3 

O  Left's  Next  move 


OLeft 

0  Righ 

62 

103 

X2 

61 

XI 

1165 

02 

01 

1512 

02 

01 

Xi 

X  Left 

X  Right 
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Because  the  lower  left  cell  has  the  highest  point  total  it  is 
selected  as  player  0  Left’s  move  for  turn  three. 

The  same  procedures  are  used  with  each  player’s  unique  set  of 
three  game  plans  to  select  players  O  Right,  X  Left,  and  X  Right  next 
moves. 
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AFFXNDXX  B 


Randomization  pattern  used  in  student  play  of  T4. 

Games  were  played  across  the  game  session  columns  and  down  the 
game  number  rows  for  each  session.  After  each  session,  data 
was  evaluated  to  try  and  adjust  the  next  session  so  that  data 
points  for  the  different  scenarios  would  be  similar.  This  did 
not  always  succeed  due  to  unequal  game  play  times  and  software 
problems . 


I 

II 

III 

IV 

V 

VI 

# 

Y 

Z 

Y 

Z 

Y 

Z 

Y 

Z 

Y 

Z 

Y 

Z 

1 

A2 

A1 

KA 

B1 

Al 

Bl 

B4 

Bl 

A2 

Bl 

B3 

Al 

2 

B1 

B  A 

A 1 

A3 

B1 

A2 

Al 

B4 

Al 

B3 

Bl 

B3 

3 

B1 

A 1 

B2 

B1 

A3 

Bl 

Bl 

Bl 

Bl 

A4 

B4 

Al 

4 

B3 

B1 

B1 

B2 

Al 

A2 

Al 

B2 

B3 

Bl 

Al 

A2 

5 

B1 

A4 

A 1 

Al 

B  A 

Al 

A3 

Al 

Bl 

Al 

A3 

Bl 

6 

A1 

B3  . 

KA 

A 1 

B1 

B3 

Al 

A4 

Bl 

B4 

Bl 

A2 

7 

A2 

B1 

B1 

A3 

B3 

Al 

B2 

Al 

A2 

Al 

Al 

Bl 

8 

B2 

A 1 

B3 

Al 

B1 

A3 

A2 

Bl 

B2 

Al 

A4 

Bl 

9 

A 1 

KA 

B 1 

B3 

KA 

Al 

Al 

B3 

Al 

A4 

Al 

A3 

10 

A3 

A1 

B  A 

Al 

Al 

Al 

Bl 

A4 

A3 

Al 

B2 

Bl 

11 

A1 

B2 

A 1 

A2 

B1 

B2 

B3 

Bl 

Al 

B2 

Bl 

B2 

12 

B1 

B1 

A3 

B1 

B2 

Bl 

Bl 

Al 

Bl 

Bl 

Al 

Al 

13 

A 1 

B  A 

B1 

A2 

Al 

A3 

Bl 

B4 

Al 

B4 

A4 

Al 

M 

B4 

B1 

A 1 

B 1 

A4 

Bl 

A2 

Al 

B4 

Bl 

Bl 

A3 

13 

A 1 

B1 

B  A 

B2 

B2 

Al 

A4 

Bl 

B3 

Al 

A2 

Bl 

16 

B1 

A2 

A  1 

KA 

BI 

B4 

Al 

A3 

Bl 

A3 

Al 

B3 
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appendix  c 


Example  of  raw  game  data  from  student  play  of  T4 . 


defLCCR  I  11:09  AM  I  9/26/91 
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STUDENT  GAME  RESULTS 


GAME 

TRTMISS 

TD 

A/C 

TRTSCORE 

CTRLMISS 

CTRLSCORE 

1 

1 

1 

0 

0 

1 

1 

2 

2 

1 

1 

0 

2 

1 

3 

2 

0 

0 

2 

1 

1 

4 

2 

0 

1 

0 

2 

1 

5 

1 

1 

1 

0 

2 

1 

6 

2 

0 

1 

0 

1 

1 

7 

1 

1 

1 

0 

2 

2 

8 

1 

0 

1 

2 

1 

0 

9 

2 

1 

0 

1 

2 

0 

10 

2 

1 

0 

0 

2 

1 

11 

1 

0 

0 

0 

1 

1 

12 

1 

1 

1 

0 

1 

2 

13 

2 

0 

1 

0 

2 

1 

14 

2 

1 

1 

0 

1 

2 

15 

1 

1 

0 

0 

2 

2 

16 

2 

1 

0 

1 

1 

1 

17 

1 

1 

1 

0 

1 

1 

18 

2 

1 

0 

1 

1 

0 

19 

2 

0 

0 

1 

2 

0  I 

20 

1 

1 

0 

1 

1 

1  1 

21 

1 

0 

1 

2 

2 

1 

22 

2 

1 

0 

2 

2 

o  ; 

23 

2 

1 

1 

1 

1 

1  ;• 

24 

2 

0 

1 

0 

2 

1  | 

25 

1 

0 

1 

0 

2 

1  i! 

26 

2 

1 

0 

0 

2 

1 

27 

2 

0 

1 

0 

1 

0 

28 

1 

0 

1 

2 

2 

1 

29 

2 

1 

1 

1 

1 

2 

30 

2 

1 

1 

0 

2 

2 

31 

1 

1 

0 

1 

2 

1 

32 

1 

0 

0 

2 

2 

1 

33 

1 

0 

1 

2 

1 

0 

34 

1 

1 

1 

1 

2 

1 

35 

1 

0 

0 

0 

2 

1 

36 

1 

1 

0 

0 

2 

2 

37 

2 

0 

1 

1 

1 

0 

38 

1 

0 

1 

0 

2 

i  !j 

39 

1 

1 

1 

0 

1 

i  | 

40 

2 

1 

1 

0 

2 

2  II 
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STUDENT  GAME  RESULTS  (continued) 


!  GAME 

TRTMISS 

TD 

A/C 

TRTSCORE 

CTRLMISS 

41 

2 

1 

1 

1 

1 

42 

2 

0 

0 

0 

2 

43 

2 

1 

0 

0 

1 

44 

1 

0 

1 

2 

1 

45 

1 

0 

0 

1 

1 

46 

2 

0 

1 

2 

1 

47 

1 

1 

1 

0 

2 

48 

2 

0 

1 

1 

2 

49 

1 

0 

0 

0 

2 

50 

1 

0 

1 

0 

1 

51 

2 

0 

1 

2 

2 

52 

1 

0 

0 

1 

2 

53 

1 

1 

1 

1 

2 

54 

2 

1 

0 

0 

2 

55 

1 

1 

0 

0 

1 

56 

2 

1 

0 

1 

1 

57 

2 

1 

0 

1 

1 

58 

2 

0 

0 

1 

2 

59 

1 

1 

0 

0 

1 

60 

1 

1 

1 

0 

2 

61 

1 

1 

0 

0 

1 

62 

2 

0 

1 

1 

1 

63 

1 

0 

1 

0 

2 

64 

2 

0 

0 

1 

1 

i  65 

1 

0 

1 

1 

1 

66 

1 

1 

1 

1 

1 

67 

1 

0 

1 

1 

1 

1  68 

2 

1 

1 

1 

1 

1  69 

2 

1 

1 

0 

2 

70 

2 

0 

1 

1 

1 

71 

1 

1 

0 

2 

2 

72 

1 

1 

1 

0 

1 

73 

1 

1 

0 

1 

2 

74  . 

2 

1 

1 

1 

1 

75 

1 

1 

0 

1 

1 

Table  coding: 

GAME  =  the  game  number  played 

in 

the  design 

sequence 

TRTMISS  =  the  treatment  team's  mission  type 
TD  =  the  level  of  tactical  delay 


A/C  =  the  level  of  area /communication  delay 
TRTSCORE  =  the  treatment  team's  mission  score 
CTRLMISS  =  the  control  team's  mission  type 
CTRLSCORE  =  the  control  team's  mission  score 


CTRLSCORE 


a 
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APPENDIX  E 


•  A  GLM  was  calculated  on  the  different  parameters  of  the 
student  gained  experiment . 

MTB  >  GLM  C6=C3  C4  C5  C7  C3*C4  C3*C5 


Factor  Levels  Values 
TRTMISS  212 

TD  2  0  1 

A/C  201 

OPPMISS  212 


Analysis  of  Variance  for  TRTSCORE 


Source 

n 

DF 

Seq  SS 

Adj  SS 

Adj  MS 

F 

r 

TRTMISS 

0.780 

1 

0.0534 

0.0394 

0.0394 

0.08 

TD 

0.019 

1 

2.6452 

2.9227 

2.9227 

5.82 

A/C 

0.465 

1 

0.1920 

0.2711 

0.2711 

0.54 

OPPMISS 

0.184 

1 

0.8771 

0.9038 

0.9038 

1.80 

TRTMISS *TD 
0.430 

1 

0.4443 

0.3166 

0.3166 

0.63 

TRTMISS*A/C 

0.436 

1 

0.3087 

0.3087 

0.3087 

0.61 

Error 

Total 

68 

74 

34.1459 

38.6667 

34.1459 

0.5021 
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Term 

Coe  .  £ 

Constant 

TRTMISS 

0.70125 

1 

TD 

-0.02346 

0 

A/C 

0.20222 

0 

OPPMISS 

0.06193 

1 

TRTMISS *TD 

0.11035 

1  0 

TRTMISS*A/C 

0.06647 

1  0 

-0.06596 

St  dev 
0.08371 

t-value 

8.38 

P 

0.000 

0.08374 

-0.28 

0.780 

0.08382 

2.41 

0.019 

0.08429 

0.73 

0.465 

0.08225 

1.34 

0.184 

0.08372 

0.79 

0.430 

0.08412 

-0.78 

0.436 

Unusual  Observations  for  TRTSCORE 


Obs 

.  TRTSCORE 

Fit 

Stdev.Fit 

Residual 

St .Resid 

22 

2.00000 

0.60651 

0.21565 

1.39349 

2.06R 

51 

2.00000 

0.62222 

0.21213 

1.37778 

2.04R 

71 

2.00000 

0.29471 

0.20618 

1.70529 

2.52R 

R  i 

denotes  an 

obs.  with 

a  large  st 

.  resid. 

•  Mann-Whitney  Confidence  Interval  and  Test  on  student 
treatment  team's  score  with  and  without  tactical  delay. 

MTB  >  mann  ell  cl2 

Mann-Whitney  Confidence  Interval  and  Test 


Cll  N  =  33  MEDIAN  = 

C12  N  =  42  MEDIAN  = 

POINT  ESTIMATE  FOR  ETA1-ETA2  IS 
95.1  PCT  C.I.  FOR  ETA1-ETA2  IS  <  -0.0000,  0.9999) 

W  =  1426.0 

TEST  OF  ETAl  =  ETA2  VS.  ETAl  N.E.  ETA2  IS  SIGNIFICANT  AT 
0.0672. 

The  test  is  significant  at  0.0451  (adjusted  for  ties) 


1.0000 
0.0000 
-0.0000 
(  -0.0000, 


•  The  correlation  between  the  treatment  team's  tactical 
delay  and  score  was  calculated. 

MTB  >  CORR  C4  C6 

Correlation  of  TD  and  TRTSCORE  =  -0.262 
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APPENDIX  P 


•  Automated  pattern  and  position  programming. 


Figure  1.  Games  A1A1,  A1A2,  A1A3,  A1A4 


Figure  2.  Games  A1B1,  A1B2,  A1B3,  A1B4 
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Figure  4.  Gaines  B1B1,  B1B2,  B1B3,  B1B4 
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APPENDIX  0 


The  following  pages  contain  the  output  of  the  Automated  T4 
game  in  an  Excel  spreadsheet  format . 

Each  game  type  is  identified  at  the  top  of  the  page.  For 
example  the  code  dfblb2  represents  a  data  file  output  in 
which  the  game  played  was  blb2- -control  team  mission, 
victory  left  and  victory  crossover;  control  team  delay, 
none;  treatment  team  mission  victory  left/victory  crossover; 
treatment  team  delay,  tactical  delay  of  one  move. 

Each  game  has  eleven  pages  of  data.  On  the  third  page  of 
every  game  type  will  appear  two  score  total  columns — one  for 
the  O  team  and  one  for  the  X  team.  The  cell  after  the  last 
entry  of  each  column  contains  the  total  sum  of  the  scores . 
The  cell  below  the  sum  contains  the  mean  score;  and  the  last 
cell  contains  the  standard  deviation  of  the  mean. 


Game  coding; 

A  =  mission  of  victory  left  and  victory  right 
B  =  mission  of  victory  left  and  victory  crossover 

1  =  no  delays 

2  =  tactical  delay  of  one  move  (direct  opponent's  moves) 

3  =  area  and  communication  delay  of  one  move  (partner's 

moves  and  partner's  opponent's  moves) 

4  =  tactical,  area  and  communication  delay  of  one  move 
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dfalal 


Game 

Game 

D 

Date 

Time 

Turn 

rmrora-i 

4/8/92 

11:15  AM 

RRIMil;] 

4/8/92 

11:18  AM 

mniddd 

4/8/92 

11:20  AM 

tmmim 

4/8/92 

11:23  AM 

raniiiWii 

4/8/92 

11:26  AM 

tmmw* 

4/8/92 

11:29  AM 

r.mminnd 

4/8/92 

11:31  AM 

tmmim 

4/8/92 

11:34  AM 

4/8/92 

11:37  AM 

rmnii.W:! 

4/8/92 

11:40  AM 

4/8/92 

11:42  AM 

4/8/92 

11:45  AM 

4/8/92 

11:48  AM 

4/8/92 

11:51  AM 

4/8/92 

11:53  AM 

msmbm 

4/8/92 

11:56  AM 

4/8/92 

11:59  AM 

gmalal  [  DDR 

4/8/92 

12:01  PM 

gmalal  [  DDR 

4/8/92 

12:04  PM 

gmalal  [  DDR 

4/8/92 

12:06  PM 

gmalal  [  DDR 

4/8/92 

12:09  PM 

gmalal  [  DDR 

4/8/92 

12:12  PM 

gmalal  [  DDR 

4/8/92 

12:14  PM 

gmalal  [  DDR 

4/8/92 

12:16  PM 

12:19  PM 

gmalal  [  DDR 

4/8/92 

12:22  PM 

gmalal  [  DDR 

4/8/92 

12:25  PM 

gmalal  [DDR,  4/8/92 

12:28  PM 

gmalal  [  DDR  4/8/92 

12:30  PM 

gmalal  [  DDR 

4/8/92 

12:33  PM 

Page  1 


Missions 

Ml 

M2 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL 

vr 

VL _ , 

XL _ 

dfalal 


Page  2 


Page  3 


dial  a  1 


Page  4 


dfalal 


Page  5 


dfalal 


Page  6 


dfalal 


Page  7 


dfalal 


Page  8 


o 


Pe 


dfalal 


Page  10 


PHSHH33E3I 

rmnsum 

naggiEsgii 

ranra 

rmswB 

mSBMEM 


imalal  [  DDR 


ri,,Ullll»l»l;'l 

mn.].];<i 

r„IU 

giimiiur.r-ii 

r„l  II  111.1.1:1 

llllll.|.J:ll 

PH3HI1333I 

l.„IU  111.1.1:1 

hm  iii.i.i:ii 

rmHii.].];ii 

mum 

tmnsEM 

BSBBnSEB 

mum 

mmm 

mum 


Date 

Time 

Turn 

Ml 

M2 

4/8/92 

12:36  PM 

8 

VL 

vr 

4/8/92 

12:39  PM 

8 

VL 

vr 

4/8/92 

12:41  PM 

8 

VL 

vr 

4/8/92 

12:44  PM 

8 

VL 

vr 

4/8/92 

12:47  PM 

8 

VL 

vr 

4/8/92 

12:49  PM 

9 

VL 

vr 
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APPENDIX  N 


•  A  GLM  was  calculated  on  the  different  parameters  of  the 
automated  experiment. 

MTB  >  GLM  C6=C3  C4  C5  C3*C4  C3*C5 
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DF 

Seq  SS 

Adj  SS 

Adj  MS 

F 

r 

TRTMISS 

.253 

1 

0.5333 

0.5333 

0.5333 

1.31 

TD 

.000 

1 

7.5000 

7.5000 

7.5000 

18.45 

A/C 

0.000 

1 

0.0000 

0.0000 

0.0000 

0.00 

TRTMISS *TD 
0.886 

1 

0.0083 

6.0083 

0.0083 

0.02 

TRTMISS*  A/C 
0.015 

1 

2.4083 

2.4083 

2.4083 

5.92 

Error 

Total 

474 

479 

192.6750 

203.1250 

192.6750 

0.4065 

Unusual  Observations  for  TRTSCORE 


Obs . 

TRTSCORE 

Fit 

Stdev.Fit 

Residual 

St .Resid 

49 

2.00000 

0.57917 

0.07128 

1.42083 

2.24R 

59 

2.00000 

0.57917 

0.07128 

1.42083 

2.24R 

109 

2.00000 

0.72083 

0.07128 

1.27917 

2 . 02R 

229 

2.00000 

0.65417 

0.07128 

1.34583 

2.12R 

338 

2.00000 

0.65417 

0.07128 

1.34583 

2.12R 

343 

2.00000 

0.65417 

0.07128 

1.34583 

2.12R 

348 

2.00000 

0.65417 

0.07128 

1.34583 

2 . 12R 

358 

2.00000 

0.65417 

0.07128 

1.34583 

2.12R 

434 

2.00000 

0.72083 

0.07128 

1.27917 

2 . 02R 

437 

2.00000 

0.72083 

0.07128 

1.27917 

2.02R 

446 

2.00000 

0.72083 

0.07128 

1.27917 

2.02R 

R  denotes  an  obs 

i.  with  a 

large  st. 

resid. 
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•  Mann-Whitney  Confidence  Interval  and  Test  of  automated 
treatment  team's  score  with  and  without  tactical  delay. 

MTB  >  mann-whitney  ell  cl2 

Cll  N  =  240  Median  =  1.0000 

C12  N  =  240  Median  =  1.0000 

Point  estimate  for  ETA1-ETA2  is  -  0.0000 

95.0  pet  c.i.  for  ETA1-ETA2  is  (-0.0000,-0.0000) 

W  =  63337.5 

Test  of  ETA1  =  ETA2  vs.  ETAl  n.e.  ETA2  is  significant  at 
0.0002 

The  test  is  significant  at  0.0000  (adjusted  for  ties) 


•  A  correlation  was  calculated  between  tactical  delay  and 
treatment  team's  score. 

MTB  >  CORR  C4  C6 

Correlation  of  TD  and  TRTSCORE  =  -0.192 

•  A  correlation  was  calculated  between  (treatment  team's 
mission  and  area/communication  delay)  and  treatment 
team's  score. 

MTB  >  CORR  C3*C5  C6 

TRTMISS  A/C 

A/C  0.000 

TRTSCORE  0.051  0.000 
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••  Calculation  of  the  overall  means  of  automated  and 
student  game  scores.  Treatment  and  control  team  scores 
are  put  together. 

MTB  >  unstack  c8  cl3  cl4; 

SUBC>  subs  clO . 

MTB  >  info 


COLUMN 

NAME 

COUNT 

Cl 

TEAMS 

555 

C2 

GAME 

555 

C3 

TRTMISS 

555 

C4 

TD 

555 

C5 

A/C 

555 

C6 

TRTSCORE 

555 

C7 

CTLMISS 

555 

C8 

CTLSCORE 

555 

C9 

CODE 

555 

CIO 

S/A 

555 

Cll 

AUTO  (T) 

480 

C12 

STUD  (T) 

75 

C13 

AUTO(C) 

480 

C14 

STUDIC) 

75 

Automated  treatment  and  control  scores  are  put  together. 

MTB  >  stack  ell  cl3  cl5 
MTB  >  info 


COLUMN 

NAME 

COUNT 

Cl 

TEAMS 

555 

C2 

GAME 

555 

C3 

TRTMISS 

555 

C4 

TD 

555 

C5 

A/C 

555 

C6 

TRTSCORE 

555 

C7 

CTLMISS 

555 

C8 

CTLSCORE 

555 

C9 

CODE 

555 

CIO 

S/A 

555 

Cll 

AUTO (T) 

480 

C12 

STUD  (T) 

75 

C13 

AUTO (C) 

480 

C14 

STUDIC) 

75 

C15 

AUTOIT+C) 

960 
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Student  treatment  and  control  scores  are  put  together. 

MTB  >  stack  cl2  cl4  cl6 
MTB  >  info 


COLUMN 

NAME 

COUNT 

Cl 

TEAMS 

555 

C2 

GAME 

555 

C3 

TRTMISS 

555 

C4 

TD 

555 

C5 

A/C 

555 

C6 

TRTSCORE 

555 

C7 

CTLMISS 

555 

C8 

CTL SCORE 

555 

C9 

CODE 

555 

CIO 

S/A 

555 

Cll 

AUTO  (T) 

480 

C12 

STUD (T) 

75 

C13 

AUTO(C) 

480 

C14 

STUD  (C) 

75 

C15 

AUTO  ( T+C ) 

960 

C16 

STUD  (T+C) 

150 

The  means  are  calculated. 
MTB  >  desc  cl5 


N 

MEAN 

MEDIAN 

TRMEAN 

STDEV 

SEMEAN 

C15 

960 

0.9719 

1.0000 

0.9688 

0.6821 

0.0220 

MIN 

MAX 

Q1 

Q3 

C15 

0.0000 

2.0000 

1.0000 

1.0000 

MTB  >  desc 

cl6 

N 

MEAN 

MEDIAN 

TRMEAN 

STDEV 

SEMEAN 

C16 

150 

0.8467 

1.0000 

0.8284 

0.7211 

0.0589 

MIN 

MAX 

Q1 

Q3 

C16 

0.0000 

2.0000 

0.0000 

1.0000 

246 


•  A  two  sample  t-test  is  performed  on  the  means. 

MTB  >  twosample  cl5  cl6 

TWOSAMPLE  T  FOR  C15  VS  C16 

N  MEAN  STDEV  SE  MEAN 

C15  960  0.972  0.682  0.022 

C16  150  0.847  0.721  0.059 

95  PCT  Cl  FOR  MU  C15  -  MU  C16:  (0.001,  0.249) 

TTEST  MU  C15  =  MU  C16  (VS  NE)  :  T=  1.99  P=0.048  DF=  192 


•  Calculating  the  mean  of  treatment  team's  score: 
automated  vs.  student. 

MTB  >  info 


COLUMN 

NAME 

COUNT 

Cl 

TEAMS 

555 

C2 

GAME 

555 

C3 

TRTMISS 

555 

C4 

TD 

555 

C5 

A/C 

555 

C6 

TRTSCORE 

555 

C7 

CTLMISS 

555 

C8 

CTLSCORE 

555 

C9 

CODE 

555 

CIO 

S/A 

555 

Cll 

AUTO (T) 

480 

C12 

STUD ( T ) 

75 

Comparison  of  means  by  two  sample  T-test . 

MTB  >  twos  ell  cl2 

TWOSAMPLE  T  FOR  AUTO  VS  STUD 

N  MEAN  STDEV  SE  MEAN 

AUTO  480  0.812  0.651  0.030 

STUD  75  0.667  0.723  0.083 

95  PCT  Cl  FOR  MU  AUTO  -  MU  STUD:  (-0.030,  0.322) 

TTEST  MU  AUTO  =  MU  STUD  (VS  NE)  :  T=  1.65  P=0.10  DF=  93 
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•  Calculating  the  mean  of  conciol  team's  score:  automated 
vs .  student . 

MTB  >  unstack  c8  cl3  cl4; 

SUBC>  subs  clO. 

MTB  >  info 


COLUMN 

NAME 

COUNT 

Cl 

TEAMS 

555 

C2 

GAME 

555 

C3 

TRTMISS 

555 

C4 

TD 

555 

C5 

A/C 

555 

C6 

TRTSCORE 

555 

C7 

CTLMISS 

555 

C8 

CTL SCORE 

555 

C9 

CODE 

555 

CIO 

S/A 

555 

Cll 

AUTO  (T) 

480 

C12 

STUD  (T) 

75 

C13 

AUTO(C) 

480 

C14 

STUD ( C ) 

75 

Comparison  of  the  means  using  the  two  sample  T-test. 

MTB  >  twos  cl3  cl4 

TWOSAMPLE  T  FOR  C13  VS  C14 

N  MEAN  STDEV  SE  MEAN 

C13  480  1.131  0.676  0.031 

C14  75  1.027  0.677  0.078 

95  PCT  Cl  FOR  MU  C13  -  MU  C14:  (-0.062,  0.271) 

TTEST  MU  C13  =  MU  C14  (VS  NE)  :  T=  1.24  P=0.22  DF=  98 
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Chi-square  calculation. 


MTB  >  info 


COLUMN 

NAME 

COUNT 

Cl 

TEAMS 

555 

C2 

GAME 

555 

C3 

TRTMISS 

555 

C4 

TD 

555 

C5 

A/C 

555 

C6 

TRTSCORE 

555 

C7 

CTLMISS 

555 

C8 

CTLSCORE 

555 

C9 

CODE 

555 

CIO 

S/A 

555 

All  treatment  team  scores  are  divided  out  into  a  table  by 
automated  or  student  and  by  score  value.  A  Chi-square  is 
then  calculated. 


MTB  >  table  clO  c6; 
SUBC>  chisquare. 


ROWS:  S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

• 

ALL 

0  155 

260 

65 

480 

1  36 

28 

11 

75 

ALL  191 

288 

76 

555 

CHI-SQUARE  = 

8.253 

WITH  D.F. 

= 

CELL  CONTENTS 

— 

COUNT 

MTB  >  table  clO 
SUBC>  chisquare. 

c8; 

ROWS:  S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0  82 

253 

145 

480 

1  16 

41 

18 

75 

ALL  98 

294 

163 

555 

CHI-SQUARE  = 

1.562 

WITH  D.F. 

= 

CELL  CONTENTS 

_  — 

COUNT 
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•  Determination  of  means  for  interaction  plot. 

Automated  treatment  score  data  was  broken  out  by  scenario 
type  to  determine  the  mean  points  to  be  used  in  the 
interaction  plot . 

(The  code  numbering  is  treatment  team  mission,  tactical 
delay,  area/communication  delay,  control  team  mission. ) 

MTB  >  info 


COLUMN 

NAME 

COUNT 

Cl 

TEAMS 

480 

C2 

GAME 

480 

C3 

TRTMISS 

480 

C4 

TD 

480 

C5 

A/C 

480 

C6 

TRTSCORE 

480 

C7 

CTLMISS 

480 

C8 

CTLSCORE 

480 

C9 

CODE 

480 

CIO 

S/A 

480 

Cll 

1001 

30 

C12 

1002 

30 

C13 

1011 

30 

C14 

1012 

30 

C15 

1101 

30 

C16 

1102 

30 

C17 

1111 

30 

C18 

1112 

30 

C19 

2001 

30 

C20 

2002 

30 

C21 

2011 

30 

C22 

2012 

30 

C23 

2101 

30 

C24 

2102 

30 

C25 

2111 

30 

C26 

2112 

30 

The  individual  scenarios  were  then  stacked  into  four  groups: 
mission  A,  no  area/comm  delay;  mission  A,  area/comm  delay; 
mission  B,  no  area/comm  delay;  and,  mission  B,  area/comm 
delay.  The  last  column  number  is  the  empty  column  into 
which  the  scores  were  stacked. 


MTB 

> 

stack 

cll 

cl2 

cl5 

cl6 

c28 

MTB 

> 

stack 

cl3 

cl4 

cl7 

cl8 

c29 

MTB 

> 

stack 

cl9 

c20 

c23 

c24 

c30 

MTB 

> 

stack 

c21 

c22 

c25 

c26 

c31 
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MTB  >  info 

COLUMN 

NAME 

COUNT 

A  Cl 

TEAMS 

480 

C2 

GAME 

480 

C3 

TRTMISS 

480 

C4 

TD 

480 

C5 

A/C 

480 

C6 

TRTSCORE 

480 

C7 

CTLMISS 

480 

C8 

CTLSCORE 

480 

C9 

CODE 

480 

CIO 

S/A 

480 

Cll 

1001 

30 

C12 

1002 

30 

C13 

1011 

30 

C14 

1012 

30 

C15 

1101 

30 

C16 

1102 

30 

C17 

1111 

30 

C18 

1112 

30 

C19 

2001 

30 

C20 

2002 

30 

C21 

2011 

30 

C22 

2012 

30 

C23 

2101 

30 

C24 

2102 

30 

C25 

2111 

30 

C26 

2112 

30 

C28 

120 

C29 

120 

C30 

120 

C31 

120 

The  mean  and  standard  deviation  was  then  calculated  for  each 
column. 


MTB  >  desc  c28  (Mission  A,  no  area/comm  delay) 


N 

MEAN 

MEDIAN 

TRMEAN 

STDEV 

SEME  AN 
C28 

120 

0.7083 

1.0000 

0.6759 

0.6533 

0.0596 

MIN 

MAX 

Q1 

Q3 

C28 

0.0000 

2.0000 

0.0000 

1.0000 
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MTB  >  desc  c29  (Mission  A,  area/comm  delay) 


SEME AN 

N 

MEAN 

C29 

0.0588 

120 

0.8500 

MIN 

MAX 

C29 

0.0000 

2.0000 

MTB  >  desc 

c30  (Mission  B, 

SEME AN 

N 

MEAN 

C30 

0.0562 

120 

0.9167 

MIN 

MAX 

C30 

0.0000 

2.0000 

MTB  >  desc 

c31  (Mission  B, 

SEME AN 

N 

MEAN 

C31 

0.0620 

120 

0.7750 

MIN 

MAX 

C31 

0.0000 

2.0000 

MEDIAN 

TRMEAN 

STDEV 

1.0000 

0.8333 

0.6437 

Q1 

0.0000 

Q3 

1.0000 

no  area/comm  delay) 

MEDIAN 

TRMEAN 

STDEV 

1.0000 

0.9074 

0.6161 

Q1 

1.0000 

Q3 

1.0000 

area/communication  delay) 

MEDIAN 

TRMEAN 

STDEV 

1.0000 

0.7500 

0.6794 

Q1 

0.0000 

Q3 

1.0000 
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APPENDIX  I 


•  Determination  of  individual  score  distributions  by 
scenario. 

Game  data  was  broken  out  by  scenario  and  by  game  type 
(student  or  automated)  and  the  score  percentages  calculated. 

MTB  >  info 


COLUMN 

NAME 

COUNT 

Cl 

TEAMS 

555 

C2 

GAME 

555 

C3 

TRTMISS 

555 

C4 

TD 

555 

C5 

A/C 

555 

C6 

TRTSCORE 

555 

C7 

CTLMISS 

555 

C8 

CTLSCORE 

555 

C9 

CODE 

555 

CIO 

S/A 

555 

Cll 

AUTO 

480 

C12 

STUD 

75 

CONSTANTS  USED:  NONE 

--Break  out  of  treatment  scores  by  scenario  and  automated  or 
student  game  type. 


MTB  >  table  clO  c6  c9; 
SUBC>  rowp. 

CONTROL:  CODE  =  1001 


ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

43.33 

43.33 

13.33 

100.00 

1 

50.00 

50.00 

100.00 

ALL 

43.75 

43.75 

12.50 

100.00 
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CONTROL:  CODE 

=  1002 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

13.33 

63.33 

23.33 

100.00 

1 

50.00 

25.00 

25.00 

100.00 

ALL 

17.65 

58.82 

23.53 

100.00 

CONTROL:  CODE 

=  1011 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

30.00 

60.00 

10.00 

100.00 

1 

16.67 

33.33 

50.00 

100.00 

ALL 

27.78 

55.56 

16.67 

100.00 

CONTROL:  CODE 

=  1012 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

30.00 

36.67 

33.33 

100.00 

1 

60.00 

— 

40.00 

100.00 

ALL 

34.29 

31 . 43 

34.29 

100.00 

CONTROL:  CODE 

=  1101 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

56.67 

36.67 

6.67 

100.00 

1 

66.67 

33.33 

100.00 

ALL 

58.33 

36.11 

5 . 56 

100.00 

CONTROL:  CODE 

=  1102 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

46.67 

53.33 

100.00 

1 

40.00 

40.00 

20.00 

100.00 

ALL 

45.71 

51.43 

2.86 

100.00 

CONTROL:  CODE 

=  1111 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

36.67 

60.00 

3.33 

100.00 

1 

80.00 

20.00 

100.00 

ALL 

42.86 

54.29 

2.86 

100.00 

/ 

/_ .  •* 

I 
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CONTROL:  CODE 

=  1112 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

20.00 

70.00 

10.00 

100.00 

1 

66.67 

33.33 

100.00 

ALL 

27.78 

63.89 

8.33 

100.00 

CONTROL:  CODE 

=  2001 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

3.33 

76.67 

20.00 

100.00 

1 

50.00 

50.00 

100.00 

ALL 

3.12 

75.00 

21.87 

100.00 

CONTROL:  CODE 

=  2002 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

33.33 

53.33 

13.33 

100.00 

1 

33.33 

66.67 

100.00 

ALL 

33.33 

54.55 

12 . 12 

100.00 

CONTROL:  CODE 

=  2011 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

• 

0 

1 

2 

ALL 

0 

3.33 

76.67 

20.00 

100.00 

1 

33.33 

50.00 

16.67 

100.00 

ALL 

8.33 

72.22 

19.44 

100.00 

CONTROL:  CODE 

=  2012 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

46.67 

33.33 

20.00 

100.00 

1 

60.00 

20.00 

20.00 

100.00 

ALL 

48.57 

31.43 

20.00 

100.00 

CONTROL:  CODE 

=  2101 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

86.67 

13.33 

100.00 

1 

20 . 00 

80.00 

100.00 

ALL 

2.86 

85.71 

11.43 

100.00 
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CONTROL:  CODE  =2102 


ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2  ' 

ALL 

0 

56.67 

30.00 

13.33 

100.00 

1 

60.00 

20.00 

20.00 

100.00 

ALL 

57.14 

28.57 

14.29 

100.00 

CONTROL:  CODE 

=  2111 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

20.00 

66.67 

13.33 

100.00 

1 

16.67 

83.33 

100.00 

ALL 

19.44 

69.44 

11.11 

100.00 

CONTROL:  CODE 

=  2112 

ROWS: 

S/A 

COLUMNS : 

TRTSCORE 

0 

1 

2 

ALL 

0 

76.67 

20.00 

3.33 

100.00 

1 

100.00 

-- 

100.00 

ALL 

79.41 

17.65 

2 . 94 

100.00 

— Break  out  of  control  scores  by  scenario  and  automated  or 
student  game  type. 

MTB  >  table  clO  c8  c9; 

JBC>  rowp . 

CONTROL:  CODE  =  1001 


ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

43.33 

30.00 

26.67 

100.00 

1 

100.00 

— 

100.00 

ALL 

40.62 

34.37 

25.00 

100.00 

CONTROL:  CODE 

=  1002 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

-- 

63.33 

36.67 

100.00 

1 

-- 

75.00 

25.00 

100.00 

ALL 

-- 

64.71 

35.29 

100.00 

CONTROL:  CODE 

=  1011 

ROWS : 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

53.33 

36.67 

10.00 

100.00 

1 

66.67 

33.33 

100.00 

ALL 

55.56 

36.11 

8.33 

100.00 

CONTROL:  CODE 

=  1012 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

-- 

90.00 

10.00 

100.00 

1 

— 

80.00 

20.00 

100.00 

ALL 

— 

88.57 

11.43 

100.00 

CONTROL:  CODE 

=  llul 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

16.67 

63.33 

20.00 

100.00 

1 

16.67 

66.67 

16.67 

100.00 

ALL 

16.67 

63.89 

19.44 

100.00 

CONTROL:  CODE 

=  1102 

ROWS : 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

53.33 

46.67 

100.00 

1 

20.00 

20.00 

60.00 

100.00 

ALL 

2.86 

48.57 

48.57 

100.00 

CONTROL:  CODE 

=  1111 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

20.00 

60.00 

20.00 

100.00 

1 

— 

60.00 

40.00 

100.00 

ALL 

17 . 14 

60.00 

22.86 

100.00 

CONTROL:  CODE 

=  1112 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

60.00 

40.00 

100.00 

1 

66.67 

33.33 

100.00 

ALL 

— 

61.11 

38.89 

100.00 
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CONTROL:  CODE  =  2001 


ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

16.67 

60.00 

23.33 

100.00 

1 

50.00 

50.00 

100.00 

ALL 

15 . 62 

59.37 

25.00 

100.00 

CONTROL:  CODE 

=  2002 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

36.67 

53.33 

10.00 

100.00 

1 

100.00 

100.00 

ALL 

42.42 

48.48 

9.09 

100.00 

CONTROL:  CODE 

=  2011 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

13.33 

53.33 

33.33 

100.00 

1 

50.00 

50.00 

100.00 

ALL 

19.44 

52.78 

27.78 

100.00 

CONTROL:  CODE 

=  2012 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

33.33 

46.67 

20.00 

100.00 

1 

20.00 

80.00 

100.00 

ALL 

31.43 

51.43 

17.14 

100.00 

CONTROL:  CODE 

=  2101 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

6.67 

53.33 

40.00 

100.00 

1 

20.00 

80.00 

-- 

100.00 

ALL 

8.57 

57.14 

34.29 

100.00 

CONTROL:  CODE 

=  2102 

ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

16.67 

53.33 

30.00 

100.00 

1 

40.00 

40.00 

20.00 

100.00 

ALL 

20.00 

51.43 

28.57 

100.00 
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CONTROL:  CODE  =  2111 


ROWS: 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

3.33 

20.00 

76.67 

100.00 

1 

50.00 

50.00 

100.00 

ALL 

2.78 

25.00 

72.22 

100.00 

CONTROL:  CODE 

=  2112 

ROWS : 

S/A 

COLUMNS : 

CTLSCORE 

0 

1 

2 

ALL 

0 

13.33 

46.67 

40.00 

100.00 

1 

25.00 

75.00 

100.00 

ALL 

11.76 

44.12 

44.12 

100.00 
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TREATMENT  TEAM  SCORES 
A1A3 


TREATMENT  TEAM  SCORES 
A1B4 


TREATMENT  TEAM  SCORES 
B1A3 


CONTROL  TEAM  SCORES 
A1B2 


CONTROL  TEAM  SCORES 
B1B2 


CONTROL  TEAM  SCORES 
B1B4 
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